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SUBJECT;  Audit  Report  on  the  Acquisition  of  the  Joint  Helmet  Mounted  Cueing 
System  Report  No.  D2(X)1-103) 


We  are  providing  this  report  for  review  and  comment.  This  special  version  of 
the  report  has  been  revised  to  omit  contractor  proprietary  information.  We  considered 
management  comments  on  a  draft  of  this  report  when  preparing  the  final  report. 

DoD  Directive  7650.3  requires  that  all  recommendations  and  potential  monetary 
benefits  be  resolved  promptly.  The  Air  Force  comments  were  partially  responsive.  As 
a  result  of  management  comments,  we  made  changes  to  finings  A  and  B.  However, 
the  Air  Force  Program  Executive  Officer  did  not  concur  with  Recommendation  B.3. 
Therefore,  we  request  additional  comments  from  the  Air  Force  Program  Executive 
Officer  on  Recommendation  B.3.  by  June  18,  2001. 

We  appreciate  the  courtesies  extended  to  the  audit  staff.  For  additional 
information  on  this  report,  please  contact  Mr.  Raymond  A.  Spencer  at  (703)  604-9071 
(DSN  664-9071)  (rspencer@dodig.osd.mil)  or  Mr.  Thomas  S.  Bartoszek  at 
(703)  604-9014  (DSN  664-9014)  (tbartoszek@dodig.osd.iml).  See  Appendix  C  for  the 
report  distribution.  The  audit  team  members  are  listed  inside  the  back  cover. 
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Report  No.  D2001-103  April  18,  2001 

(Project  No.  D2000AB-0220) 

Acquisition  of  the  Joint  Helmet  Mounted  Cueing  System 
Executive  Summary 


Introduction.  The  Joint  Helmet  Mounted  Cueing  System  consists  of  a  helmet  and 
aircraft  interface  components  that  will  allow  aircraft  fighter  pilots  to  engage  and  destroy 
airborne  targets  within  visual  range  with  a  first  look,  first  shot,  first  kill  advantage 
provided  by  the  High-Off-Boresight  Capability.  The  capability  allows  Ae  pilots  to 
engage,  lock,  and  launch  weapons  at  a  target  beyond  the  range  of  the  aircraft's  radar 
and  within  the  pilot's  field  of  view.  The  capability  will  not  be  available  until  after  the 
Navy  deploys  the  AIM-9X  missile.  The  AIM-9X  is  cuwently  in  development  with  an 
estimated  completion  date  of  May  2003.  If  the  missile  is  not  deployed  at  the  same  time 
as  the  helmet,  there  will  still  be  added  capability  because  the  helmet  visor  displays  data 
needed  during  combat  such  as  airspeed,  dtitude,  target  range,  weapons,  sensors,  and 
navigation.  The  Air  Force  plans  to  employ  the  Joint  Helmet  Mounted  Cueing  System 
as  upgrades  on  the  F-15  C/D  and  F-16  C/D  blocks  40  and  50  aircraft,  and  insert  the 
helmet  into  the  production  line  for  the  F-22.  The  Navy  plans  to  incorporate  the  helmet 
in  the  F-18  E/F  production  line  and  as  a  planned  upgrade  to  the  F-18  C/D  model.  The 
Joint  Helmet  Mounted  Cueing  System  is  a  joint  Air  Force  and  Navy  Acquisition 
Category  III  program  under  Ae  milestone  decision  authority  of  the  Air  Force  Progr^ 
Executive  Officer  for  Fighter  and  Bomber  Programs.  The  helmet  is  in  the  engineering, 
manufacturing,  and  development  phase  of  the  acquisition  cycle  and  is  scheduled  for  a 
Milestone  III  fUll-rate  production  decision  in  April  2002.  As  of  December  31,  1999, 
the  estimated  total  program  cost  was  about  $672  million. 

Objectives.  The  audit  objective  was  to  evaluate  the  overall  management  of  the  Joint 
Helmet  Mounted  Cueing  System.  Specifically,  the  audit  determined  whether  the  Air 
Force  is  cost-effectively  readying  the  system  for  the  production  phase  of  the  acquisition 
process.  We  also  evaluated  the  management  control  program  as  it  related  to  the  overall 
objectives.  This  report  addresses  testing  and  evaluation  and  contracting.  A  later  report 
will  address  requirement  evolution  and  affordability  and  joint  management. 

Results.  The  Joint  Helmet  Mounted  Cueing  System  operational  tests,  as  planned, 
would  not  provide  the  objective  test  results  necessary  to  support  the  full-rate  production 
decision  in  April  2002.  As  a  result,  the  Air  Force  would  spend  about  $6  million  for 
operational  testing  without  adequately  determining  whether  the  system  will  be 
operationally  effective,  suitable,  and  would  provide  the  warfighter  with  a  first  look, 
first  shot  advantage  within  visual  range  in  the  air-to-air  combat  arena  (finding  A).  In 
addition,  the  acquisition  approach  of  the  joint  program  needed  improvement  to 
recognize  the  risks  associated  with  the  rebaseline  and  the  contracting  structure  of  the 
Joint  Helmet  Mounted  Cueing  System  program,  and  to  explore  component  breakout 
opportunities  for  full-rate  production.  As  a  result,  the  evaluation,  identification,  and 
management  of  contractor  performance  is  at  risk,  and  the  joint  program  office  would 
miss  the  opportunity  to  put  approximately  $17  million  of  funds  to  better  use  through 


purchasing  five  components  directly  from  manufacturers  and  eliminating  the  non-value- 
added  overhead  profit  of  25  percent  from  intermediate  tiers  of  contractors  (finding  B). 
See  Appendix  A  for  details  on  the  management  control  program  on  updating  controls  in 
the  operational  testing  and  the  acquisition  plan. 

Summary  of  Recommendations.  We  recommend  that  the  Air  Force  Program 
Executive  Officer  for  Fighter  and  Bomber  Programs  require  the  Joint  Helmet  Mounted 
Cueing  System  Program  Manager  to  update  the  Test  and  Evaluation  Master  Plan; 
identify  threat  threshold  and  enhancement  objective  values  of  the  system;  conduct  and 
present  a  component  breakout  study  as  exit  criteria  for  the  engineering,  manufacturing, 
and  development  phase;  revise  the  acquisition  plan;  and  establish  a  process  to  regularly 
update  the  Test  and  Evaluation  Master  Plan  and  the  Single  Acquisition  Management 
Plan.  We  also  recommend  that  the  Commander,  Air  Force  Operational  Test  and 
Evaluation  Center,  revise  the  Multi-Service  Operational  Test  and  Evaluation  Plan, 
identify  pass-and-fail  criteria  for  the  questionnaires,  and  revise  the  projection  plan. 

Management  Comments.  The  Air  Force  Program  Executive  Officer  for  Fighter  and 
Bomber  Programs  agreed  to  update  the  Test  and  Evaluation  Master  Plan,  to  conduct  a 
component  breakout  study,  and  to  revise  the  acquisition  plan.  He  did  not  concur  with 
establishing  a  process  to  update  the  acquisition  plan  because  he  stated  that  a  process 
already  exists.  In  addition,  he  believed  that  only  $5  million  of  funds  may  be  put  to 
better  use  from  component  breakout  because  the  audit  eliminated  all  contractor  profit. 
The  Commander,  Air  Force  Operational  Test  and  Evaluation  Center,  disagreed  with 
revising  the  Multi-Service  Operational  Test  and  Evaluation  Plan  to  identify  pass-and- 
fail  criteria  for  the  questionnaires  used  to  evaluate  human  factors  and  revising  the 
statistical  projection  plan.  Because  of  affordability  considerations,  he  stated  that  the 
questionnaires  were  intended  to  gather  subjective  inputs  and  that  the  Center  staff  would 
combine  the  responses  with  other  data  to  judge  the  operational  effectiveness  and 
suitability  of  the  helmet.  He  stated  that  the  Joint  Helmet  Mounted  Cueing  System 
human  factor  issues  will  be  rated  as  favorable  if  the  majority  of  the  ratings  are  positive 
and  unfavorable  if  the  majority  of  the  ratings  are  negative.  He  stated  that  additional 
interviews  will  be  conducted  in  instances  where  the  ratings  do  not  provide  a  clear 
answer.  The  complete  text  of  management  comments  is  in  the  Management  Comments 
section. 

Audit  Response.  We  consider  the  management  comments  to  be  p^ially  responsive. 

If  the  program  office  had  a  process  to  update  the  acquisition  plan,  it  would  have 
updated  it  after  the  rebaseline.  Our  calculation  of  potential  monetary  benefits  was  a 
target  amount  that  excluded  the  prime  and  subcontractor’s  profits  because  the 
components  would  be  purchased  directly  from  the  manufacturers.  We  understand  that 
the  exact  amount  of  benefits  will  not  be  known  until  the  contract  is  negotiated.  The 
Director,  Air  Force  Operational  Test  and  Evaluation  Center,  provided  an  acceptable 
alternative  plan  of  action  to  assess  the  system’s  operational  suitability  in  relation  to 
human  factors. 

We  request  that  the  Air  Force  Program  Executive  Officer  for  Fighter  and  Bomber 
Programs  provide  additional  comments  by  June  18,  2001,  on  the  establishment  of  a 
process  to  update  planning  documents. 
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Background 

The  Joint  Helmet  Mounted  Cueing  System  (JHMCS)  consists  of  a  helmet  and 
aircraft  interface  components  that  will  allow  aircraft  fighter  pilots  to  engage  and 
destroy  airborne  targets  within  visual  range  with  a  first  look,  first  shot,  first  kill 
advantage  provided  by  the  High-Off-Boresight  Capability  and  other  sensors  and 
weapons.  The  capability  allows  the  pilots  to  engage,  lock,  and  launch  weapons 
at  a  target  beyond  the  range  of  the  aircraft's  radar  and  within  the  pilot's  field  of 
view.  The  capability  will  not  be  available  until  after  the  Navy  deploys  the  AIM- 
9X  missile  currently  in  development.  The  JHMCS  also  displays  data  needed 
during  combat  such  as  airspeed,  altitude,  target  range,  weapons,  sensors,  and 
navigation  in  the  visor.  The  feature  saves  time  when  engaged  in  combat  because 
the  pilot  can  view  vital  information  in  the  visor  instead  of  on  the  cockpit  display 
panel.  The  Air  Force  plans  to  employ  the  JHMCS  as  upgrades  on  the  F-15  C/D 
and  F-16  C/D  blocks  40  and  50  aircraft,  and  insert  the  JHMCS  into  the 
production  line  for  the  F-22.  The  Navy  plans  to  incorporate  the  JHMCS  in  the 
F-18  E/F  production  line  and  as  a  planned  upgrade  to  the  F-18  C/D  model.  The 
two  aircraft  that  will  first  receive  the  JHMCS  are  the  Navy  F-18  E/F  and  the 
Air  Force  F-15  C/D  models.  The  Boeing  Company  is  the  prime  contractor  for 
the  F-18  and  the  F-15  aircraft  and  the  JHMCS.  Boeing  will  integrate  the  helmet 
into  the  aircraft.  Lockheed  Martin  is  the  manufacturer  for  the  F-16  and  F-22 
aircraft  and  will  integrate  the  helmet  into  these  aircraft. 

The  JHMCS  is  a  joint  Air  Force  and  Navy  Acquisition  Category  IE  program 
under  the  milestone  decision  authority  of  the  Air  Force  Program  Executive 
Officer  for  Fighter  and  Bomber  Programs.  The  JHMCS  is  in  the  engineering, 
manufacturing,  and  developing  phase  of  the  acquisition  cycle,  which  began  in 
January  1997  with  planned  completion  in  March  2002.  The  development 
contract  was  a  cost-plus-award-fee  instrument  for  approximately  $77  million. 
The  Program  Executive  Officer  had  scheduled  the  Milestone  III,  full-rate 
production  decision  for  September  2001,  with  operational  testing  to  begin  in 
December  1999  for  the  F-18  and  in  October  2000  for  the  F-15.  However,  in 
December  1999,  several  technical  challenges  remained  during  development 
including  system  maturity,  reliability,  and  maintenance.  The  Program 
Executive  Office  restructured  the  program  and  rescheduled  the  production 
decision  for  April  2002  with  operational  testing  to  begin  in  September  2001. 

The  restructure  extended  the  engineering,  manufacturing,  and  development 
phase  until  March  2002  permitting  time  to  solve  the  problems.  Also,  the 
JHMCS  joint  program  office  added  a  second  low-rate  initial  production  to 
commence  in  March  2001  for  the  F-15,  F-16,  F-18  E/F  and  F-22  aircraft.  The 
first  low-rate  initial  production  for  the  F-18  began  in  May  2000.  Restructure 
costs  totaled  about  $22  million.  As  of  December  31,  1999,  the  joint  program 
office  for  the  JHMCS  estimated  that  the  cost  for  developing  and  producing 
1,776  helmets  to  be  $641  million,  which  included  changes  to  1,882  aircraft. 
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Objectives 


The  audit  objective  was  to  evaluate  the  overall  management  of  the  JHMCS. 
Specifically,  the  audit  determined  whether  the  Air  Force  is  cost-effectively 
readying  the  system  for  the  production  phase  of  the  acquisition  process.  The 
audit  was  conducted  in  accordance  with  the  Inspector  General,  DoD,  critical 
program  management  element  approach.  See  Appendix  A  for  a  discussion  of 
the  audit  scope  and  methodology,  the  management  control  program,  and  prior 
audit  coverage.  This  report  addresses  testing  and  contracting.  A  later  report 
will  address  requirement  evolution  and  affordability  and  joint  management. 
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A.  Utility  of  Planned  Operation  Testing 

The  JHMCS  operational  tests,  as  planned,  would  not  provide  objective 
test  results  necessary  to  support  the  JHMCS  full-rate  production  decision 
in  April  2002  because  of  the  following: 

•  The  Operational  Requirements  Document  (ORD)  did  not 
identify  operational  parameters  and  articulate  requirements  in 
measurable  terms. 

•  The  Test  and  Evaluation  Master  Plan  (TEMP)  was  outdated 
and  insufficient  to  provide  the  overall  structure  for  an 
objective  testing  program  and  to  ensure  that  the  operational 
tests  would  provide  objective  results  that  can  determine 
whether  the  program  is  operationally  effective  and  suitable  for 
meeting  the  warfighters’  needs  in  entering  production. 

•  The  Multi-Service  Operational  Test  and  Evaluation  Plan  did 
not  include  a  baseline  threshold  or  objectives  to  measure 
success,  did  not  include  plans  for  a  valid  statistic^  projection, 
did  not  specify  pass-and-fail  criteria,  and  did  not  include  a 
confidence  level  for  questionnaires  developed  to  measure  and 
project  human  factor  elements  of  critical  operational  issues. 

As  a  result,  the  Air  Force  would  spend  about  $6  million  for  operational 
testing  without  adequately  determining  whether  the  JHMCS  will  be 
operationally  effective,  suitable,  and  provide  the  warfighter  with  a  first 
look,  first  shot  advantage  within  visual  range  in  the  air-to-air  combat 
arena. 

Testing  Criteria 

The  DoD  Regulation  5000.2-R  “Mandatory  Procedures  for  Major  Defense 
Acquisition  Programs  and  Major  Automated  Information  System  Acquisition 
Programs,”  dated  October  23,  2000,  provides  that  the  ORD  defme  the  system 
capability  needed  to  satisfy  the  mission  need  and  identify  operational 
performance  parameters  in  measurable  terms.  The  parameters  must  identify  a 
minimum  acceptable  value  (threshold)  required  to  satisfy  the  mission  need  and 
may  contain  objective  values  that  represent  a  measurable  beneficial  increase  in 
capability  above  the  threshold. 

The  Regulation  also  states  that  the  TEMP,  which  outlines  the  overall  structure 
and  the  objectives  of  the  testing  program,  provides  the  operational  testing 
program  to  evaluate  whether  the  system  is  operationally  effective  and  suitable  to 
satisfy  the  mission  need  before  the  fiill-rate  production  decision.  The  TEMP 
must  also  provide  a  framework  to  generate  detailed  test  and  evduation  plans.  In 
this  respect,  the  TEMP  must  identify  critical  operational  effectiveness  and 
suitability  issues,  measures  of  effectiveness,  and  measures  of  performance  with 
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appropriate  quantitative  criteria  to  provide  evidence  for  analysis  of  the  system. 
Program  officials  must  update  the  TEMP  at  milestones  or  when  other  events 
change  the  program  significantly. 

Operational  Requirements  Document 

The  ORD,  dated  December  18,  1996,  states  that  the  mission  need  was  to  attain  a 
first  look,  first  shot  advantage  in  air-to-air  combat  within  visual  range  over  the 
threat  identified.  The  ORD  identified  the  threat  as  the  Russian’s  advanced 
AA-1 1  air-to-air  missile  with  a  helmet-mounted  sight  employed  on  the  MIG  29 
and  SU27  fighters.  To  achieve  the  advantage,  the  ORD  provided  that  Ae  new 
system  should  enhance  the  capabilities  of  aircrews.  The  ORD  did  not  identify 
the  system  functions  in  defined  terms  and  did  not  address  system  performance 
characteristics  in  terms  of  minimal  acceptable  threat  thresholds,  enhancements, 
or  objectives  that  would  provide  the  advantage. 

Officials  from  the  Air  Combat  Command  who  prepared  the  ORD  stated  that  to 
apply  additional  technical  performance  criteria  would  only  increase  the  risk  of 
driving  the  scenarios  to  meet  specific  test  points,  instead  of  assessing  how  the 
helmet  enhanced  the  weapon  employment  opportunities  in  tactical  scenarios. 

The  absence  of  measurable  threat  thresholds  and  enhancement  objective  values 
may  only  result  in  enhancements  to  the  present  system  and  may  not  achieve  any 
actual  improvement  in  performance  over  the  threat.  The  ORD  does  not  define 
the  threshold  values  of  the  threat  and  objective  values  as  enhmcements  beyond  a 
threat  baseline.  Defining  those  values  will  help  ensure  that  pilots  equipped  with 
the  JHMCS  will  have  a  distinct  measurable  advantage  over  the  threat  in  air-to- 
air  combat. 

Test  and  Evaluation  Master  Plan 


The  approved  TEMP,  dated  November  27,  1996,  includes  aspects  of  the 
program  that  are  outdated  and  incomplete.  For  example,  one  platform  system 
designated  to  obtain  the  JHMCS  capability  was  the  Navy’s  AV-8B.  Officials 
from  the  joint  program  office  indicated  that  the  platform  was  a  goal  platform 
and  not  one  designated  to  receive  the  JHMCS.  In  addition,  the  Navy  did  not 
fund  the  helmet  for  the  AV-8B.  Further,  the  schedule  in  the  TEMP  showed 
production  to  occur  in  2000  and  operational  testing  to  commence  in  late  1999. 
The  TEMP  identified  measures  of  effectiveness  but  did  not  include  measures  of 
performance  or  contain  appropriate  quantitative  criteria,  such  as  threshold  and 
objective  values  to  provide  evidence  for  analysis  of  the  system,  in  part,  because 
the  ORD  did  not  provide  minimum  threshold  and  objective  values  as  a  baseline 
for  measurement.  Program  officials  stated  that  they  attempted  to  update  the 
TEMP  in  March  1998;  however,  because  the  potential  for  restructure  was 
apparent,  they  decided  to  wait  until  after  the  restructure  when  funding  decisions 
and  other  concerns  that  might  impact  testing  would  be  made.  The  restructure  in 
December  1999  rescheduled  the  production  decision  until  April  2002  and  the 
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operational  testing  until  September  2001.  Also,  the  restructure  added  a  second 
low-rate  initial  production  to  the  program.  These  changes  were  not  included  in 
the  TEMP. 

The  joint  program  office  must  update  the  TEMP  to  reflect  the  current  posmre  of 
the  program  after  the  restructure.  The  TEMP  must  identify  appropriate 
quantitative  criteria,  such  as  threshold  and  objective  values,  to  provide  evidence 
for  analysis  of  the  system,  thereby  ensuring  that  the  requirements  of  the  user  to 
achieve  a  first  look,  first  shot  capability  are  met.  The  TEMP  must  also  be  an 
effective  outline  for  the  overall  testing  structure  of  the  program  and  provide  the 
foundation  for  objective  operational  tests. 

Multi-Service  Operational  Test  and  Evaluation  Plan 


The  Multi-Service  Operational  Test  and  Evaluation  Plan  (the  Plan),  dated 
October  1999,  is  designed  to  determine  the  operational  effectiveness  and 
suitability  of  the  JHMCS  and  whether  the  system  is  ready  for  production.  To 
assist  the  testers  in  answering  the  effectiveness  and  suitability  questions,  the 
Plan  identified  the  following  four  critical  operational  issues  strucmred  in  the 
form  of  a  question,  which  Ae  testers  must  address: 

•  Does  the  JHMCS  enhance  air-to-air  weapon  emplo)mient 
opportunities  in  a  Within  Visual  Range  environment? 

•  Does  the  JHMCS  effectively  interface  and  operate  with  the  aircrew 
and  required  systems? 

•  Does  the  JHMCS  reliability  and  maintainability  support  the 
operational  tasking? 

•  Is  the  JHMCS  supportable  in  the  operational  environment? 

The  Plan  did  not  fully  detail  the  first  two  operational  issues.  The  first  issue 
involves  the  ability  of  the  JHMCS  to  enhance  the  capability  of  the  warfighter. 

To  satisfy  the  requirement,  the  Plan  proposed  a  series  of  missions  that  compared 
the  first  look,  first  shot  performance  of  like  friendly  aircraft  with  and  without 
the  JHMCS.  While  the  missions  will  provide  some  insight  into  the  JHMCS 
enhanced  capabilities,  they  will  not  establish  a  minimum  acceptable  threshold 
against  the  threat.  The  absence  of  a  minimum  threat  and  objective  values  in  the 
ORD  and  the  TEMP  contributed  to  the  testers’  decision  to  use  frien^y  aircraft 
data  to  measure  and  report  the  operational  effectiveness  and  suitability  of  the 
program. 

The  ORD  defines  the  system  capability  needed  to  satisfy  the  mission  need  and 
identifies  operational  performance  parameters  in  measurable  terms.  The 
parameters  must  identify  a  minimum  acceptable  value  (threshold)  required  to 
satisfy  the  mission  need  and  may  contain  objective  values  that  represent  a 
measurable  beneficial  increase  in  capability  above  the  threshold. 
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After  the  warfighter  defines  the  threshold  and  objective  values  in  the  ORD  and 
TEMP,  the  test  agency  must  update  the  Plan  to  reflect  the  criteria  for  measuring 
operational  effectiveness  and  suitability  of  the  JHMCS  to  enhance  air-to-air 
weapon  emplo)unent  opportunities. 

The  second  issue  addresses  human  factors  such  as  display,  comfort,  situational 
awareness,  and  day  and  night  operability.  To  help  assess  the  factors,  the 
12  pilots  who  are  desi^ated  to  fly  the  operational  testing  missions  will  complete 
a  questionnaire  regarding  their  experience  using  the  system.  The  following  two 
questions  and  answers  are  a  sample  of  those  included  in  the  questionnaire. 

1.  Rate  the  acceptability  of  the  JHMCS  comfort  on  a  “normal”  duration 
mission  (1.5  hours). 

Completely  Largely  Somewhat  Somewhat  Largely  Completely  Not 

Unacceptable  Unacceptable  Unacceptable  Acceptable  Acceptable  Acceptable  Applicable 

2.  Rate  the  acceptability  of  the  fit  of  the  JHMCS. 

Completely  Largely  Somewhat  Somewhat  Largely  Completely  Not 

Unacceptable  Unacceptable  Unacceptable  Acceptable  Acceptable  Acceptable  Applicable 

However,  the  testers  had  not  defined  pass-and-fail  criteria  for  the  answers  to  the 
questions.  Testing  officials  stated  that  they  plan  to  analyze  the  operator  rating 
of  human  factors  after  the  questionnaires  are  completed.  The  results  will  appear 
in  the  final  test  report,  allowing  the  warfighters  to  decide  whether  to  acquire  the 
system.  In  addition,  because  each  test  pilot  interaction  is  independent,  the 
testers  intend  to  use  statistical  sampling  methods  to  accurately  project  the 
operational  effectiveness  of  human  factors  to  the  universe  of  Air  Force  and 
Navy  pilots  that  are  targeted  to  be  outfitted  with  the  helmet.  The  testers  did  not 
have  a  statistical  plan  or  process  in  place  to  evaluate  the  results.  Accordingly, 
the  small  size  of  their  pilot  sample  and  its  nonrandom  nature  precludes  valid 
conclusions.  Finally,  the  Plan  does  not  identify  an  acceptable  level  of 
confidence  for  the  questionnaire. 

Considering  the  complex  nature  of  the  questionnaire,  the  lack  of  pass-and-fail 
criteria,  and  the  lack  of  a  defined  confidence  level,  the  current  method  of 
evaluation  does  not  provide  a  decision  structure  that  permits  cle^  answers  on 
the  system’s  operational  effectiveness  and  suitability  for  evaluatmg  human 
factors.  Accordingly,  the  testers  must  revise  the  Plan  to  recognize  these 
key  aspects. 

Conclusion 


The  DoD  Regulation  5000. 2-R  provides  that  the  ORD  define  the  system 
capability  needed  to  satisfy  the  mission  need  and  identify  operational 
performance  parameters  in  measurable  terms.  Threat  thresholds  and  objective 
values  must  be  defined  in  the  ORD,  the  TEMP,  and  the  Plan.  In  addition,  the 
TEMP  must  provide  a  framework  within  which  to  generate  detailed  test  and 
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evaluation  plans  that  include  critical  operational  effectiveness  and  suitability 
issues,  measures  of  effectiveness,  and  measures  of  performance  with  appropriate 
quantitative  criteria.  The  joint  program  office  must  update  the  TEMP  when 
significant  events  occur  that  change  the  program. 

The  Air  Force  will  not  have  objective  test  results  at  the  full-rate  production 
decision  because  the  ORD,  the  TEMP  and  the  Plan  did  not  define  the  threat 
threshold  and  objective  values.  Also,  the  Plan  did  not  include  pass-and-fail 
criteria,  a  valid  statistical  projection  process,  and  confidence  level  for  evaluation 
of  human  factors  through  questionnaire.  As  a  result,  the  Air  Force  will  spend 
about  $6  million  for  operational  testing  without  determining  whether  the 
JHMCS  will  be  operationally  effective,  suitable,  and  provide  the  warfighter  with 
a  first  look,  first  shot  advantage  within  visual  range  in  air-to-air  combat. 

Program  Executive  Office  for  Fighter  and  Bomber  Programs 


On  October  18,  2000,  we  met  with  senior  officials  from  the  Air  Force  Program 
Executive  Office  for  Fighter  and  Bomber  Programs  to  discuss  the  testing  issues. 
We  discussed  the  ORD,  TEMP,'  and  Plan  issues.  Senior  officials  stated  that  the 
program  would  be  completed  before  the  ORD  was  updated,  revised,  and 
approved;  therefore,  they  suggested  that  we  recommend  changes  to  the  TEMP 
and,  accordingly,  the  revisions  would  be  incorporated  in  the  Plan  when  the 
TEMP  was  revised.  In  addition,  because  the  warfighters  and  test  agencies  are 
concurrence  officials  in  the  TEMP  approval  process,  all  interested  parties  would 
validate  the  revisions  and  decisions  before  operational  testing  would  begin. 

While  they  do  not  conform  to  the  strict  interpretation  of  DoD 

Regulation  5000.2R,  the  recommended  actions  will  accomplish  the  objectives 

because  they  involve  the  testers,  the  warfighters,  and  the  joint  program  office. 

Management  Comments  on  the  Finding  and  Audit  Response 


Program  Executive  Officer  for  Fighter  and  Bomber  Programs  Comments. 

The  Program  Executive  Officer  for  Fighter  and  Bomber  Programs  did  not  agree 
that  the  ORD  failed  to  identify  operational  parameters  and  articulate 
requirements  in  measurable  terms.  He  stated  that  while  the  ORD  requires  the 
JHMCS  to  cue  the  radar,  navigation  system,  missiles  and  display  information  in 
the  visor,  the  objective  technical  requirements  or  parameters  are  included  in  the 
technical  and  engineering  documents.  Accordingly,  if  the  JHMCS  meets  these 
parameters,  pilots  will  achieve  a  first  shot  advantage.  He  further  stated  that  the 
threat  examples  identified  in  the  ORD  were  used  to  establish  objective 
evaluation  criteria.  A  failure  of  the  cueing  and  displaying  capabilities  noted 
during  testing  will  be  reflected  in  objective  measurable  readiness  and  logistics 
ratings  for  operational  suitability  and  effectiveness.  He  also  disagreed  that  the 
Multi-Service  Operational  Test  and  Evaluation  Plan  did  not  include  a  baseline 
threshold  or  objectives  to  measure  success,  did  not  include  plans  for  a  valid 
statistical  projection,  did  not  specify  pass-and-fail  criteria,  and  did  not  include  a 
confidence  level  for  questionnaires  developed  to  measure  and  project  human 
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factor  elements  of  critical  operational  issues.  He  stated  that  the  plan  was 
consistent  with  the  original  TEMP  and  contained  test  procedures  to  satisfy 
critical  operational  issues  and  to  meet  measures  of  performance.  In  addition, 
the  updated  TEMP  and  the  Plan  will  include  test  procedures  to  evaluate  the 
JHMCS  capability  against  the  threat.  The  results  will  be  included  in  the 
operational  effectiveness  assessment.  He  stated  that  responses  to  the  human 
factor  questionnaires  will  depend  on  the  judgment  of  the  testers  and  will  provide 
insight  into  the  system  characteristics  that  can  be  used  to  develop  training  or 
make  refinements  in  operational  concepts.  See  Appendix  A  for  additional 
comments  on  the  cited  management  control  deficiencies. 

Audit  Response.  The  Air  Force  Program  Executive  Officer  for  Fighter  and 
Bomber  Programs  stated  that  the  ORD  defined  the  functions  of  JHMCS.  He 
stated  that  the  operational  parameters  and  requirements  for  these  functions, 
stated  in  measurable  terms,  are  included  in  technical  and  engineering 
documents.  However,  DoD  Regulation  5000. 2-R  clearly  provides  that  the  ORD 
and  not  subsequent  engineering  documents  developed  by  the  contractor  defme 
the  system  capability  needed  to  satisfy  the  mission  need  and  identify  operational 
performance  parameters  in  measurable  terms.  The  parameters  must  identify  a 
minimum  acceptable  value  (threshold)  required  to  satisfy  the  mission  need  and 
may  contain  objective  values  that  represent  a  measurable  beneficial  increase  in 
capability  above  the  threshold.  While  engineering  documents  are  valuable  in 
defining  the  system's  actual  characteristics  in  meeting  the  requirements  of  the 
ORD,  the  ORD  must  set  the  standard  for  the  JHMCS  and  not  the  engineering 
documents.  The  warfighter  and  not  the  contractor  must  clearly  be  in  control  of 
requirements  determination  and  the  definition  for  the  system’s  capability  in 
terms  of  thresholds  and  objective  values. 

He  also  stated  that  the  Plan  was  consistent  with  the  original  TEMP  and 
contained  test  procedures  to  satisfy  critical  operational  issues  and  meet  measures 
of  performance.  However,  he  did  not  address  how  the  Plan  established  a 
minimum  acceptable  threshold  and  objective  values  against  the  threat  to  address 
the  first  two  operational  issues.  Rather,  the  Plan  does  not  include  values  or 
criteria  to  measure  operational  effectiveness  and  suitability  of  the  JHMCS  to 
enhance  air-to-air  weapons  employment  opportunities.  The  updated  Plan  should 
provide  the  needed  baseline  to  measure  success  of  critical  operational  issues. 

Air  Force  Operational  Test  and  Evaluation  Center  Comments.  The 
Commander,  Air  Force  Operational  Test  and  Evaluation  Center,  did  not  agree 
that  the  Multi-Service  Operational  Test  and  Evaluation  Plan  excluded  a  baseline 
threshold  and  objectives  to  measure  success,  excluded  plans  for  a  valid 
statistical  projection,  excluded  pass-and-fail  criteria,  and  excluded  a  confidence 
level  for  questionnaires  developed  to  measure  and  project  human  factor  elements 
of  critical  operational  issue.  He  stated  that  the  baseline  will  be  evaluated  by 
comparing  missions  with  and  without  the  JHMCS  on  friendly  aircraft.  In 
addition,  flights  will  be  made  against  threat  aircraft,  if  available,  and  the  test 
team  will  conduct  baseline  comparisons  of  threshold  aircraft  with  the  JHMCS  to 
the  threat  aircraft.  The  updated  TEMP  will  include  threat  capabilities  to  be  used 
for  the  baseline.  Concerning  the  inclusion  of  pass  and  fail  criteria  and  a 
confidence  level  for  the  subjective  human  factors,  he  stated  that  the  results 
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would  not  be  statistically  valid  for  the  small  sample  of  test  pilots.  A  valid 
statistical  projection  plan  would  require  using  a  sample  that  would  exceed 
resource  availability  for  the  benefits  derived.  The  current  sample  is  sufficient  to 
provide  clear  answers  concerning  the  impact  of  human  factors  elements  on  the 
operational  effectiveness  and  suitability  because  test  pilots  consists  of  experts 
with  extensive  experience  in  determining  military  utility  of  the  system  under 
test.  In  addition,  while  it  was  not  appropriate  to  apply  objective  pass  and  fail 
criteria  to  the  human  factors  analysis  of  the  JHMCS  a  definitive  answer 
concerning  the  effectiveness  and  suitability  of  the  human  factors  will  be 
provided.  For  each  subjective  critical  operational  issue,  measures  of 
effectiveness,  and  measures  of  performance,  the  system  will  be  rated  as 
favorable  if  the  majority  of  the  rating  is  positive  and  imfavorable  if  they  ^e 
rated  unfavorable.  Additional  interviews  will  be  conducted  where  the  rating  do 
not  provide  clear  answers.  In  addition,  descriptive  statistics  will  be  used  to 
summarize  the  data  and  a  narrative  summary  of  the  majority  strengths  and 
weaknesses  of  the  system  will  be  provided. 

Audit  Response.  The  JHMCS  must  enhance  the  capability  of  the  warfighter 
beyond  the  threat  to  allow  for  a  the  first  look,  first  shot  performance.  Although 
friendly  missions  will  provide  some  insight  into  the  JHMCS  capabilities,  they 
will  not  establish  a  minimum  acceptable  threshold  against  the  threat.  In 
addition,  the  missions  will  not  provide  an  adequate  assessment  of  the  system’s 
performance  capabilities.  By  comparing  flights  made  against  actual  threat 
aircraft  or  by  a  baseline  comparison  of  threshold  aircraft  with  the  JHMCS  to  the 
threat  aircraft,  the  enhanced  capability  of  the  system  will  be  demonstrated  and 
will  determine  if  the  helmet  provides  the  first  look,  first  shot  performance.  The 
actions  to  update  the  TEMP  to  include  the  threat  capability  to  be  used  as  a 
baseline  are  a  positive  step. 

We  agree  that  the  sample  size  will  not  be  statistically  valid  for  projectiom  based 
on  a  sample  size  of  12  pilots  and  that  funds  may  not  be  available  for  additional 
pilots.  The  mathematical  laws  dictating  the  sample  size  required  to  address  a 
question  makes  explicit  the  minimum  amount  of  information  iieeded.  A  smaller 
sampling  necessarily  collects  less  than  this  minimum  and,  logically,  it  is 
insufficient  to  provide  clear  answers.  The  two  viable  options  are  to  either 
conduct  an  adequate  sample  or,  if  resources  are  insufficient,  not  conduct  a 
sample  at  all.  Performing  an  inadequate  sampling  at  best  will  produce  results 
recognized  as  insufficient,  and  at  worst  will  yield  insufficient  results  which 
nevertheless  are  used  for  management  decision-making  purposes.  These  will 
not  provide  clear  answers  on  the  impact  of  human  factor  elements  on  the 
JHMCS  operational  effectiveness  and  suitability.  However,  the  Commander, 
Air  Force  Operational  Test  and  Evaluation  Center,  agreed  to  instimte  a  more 
objective  process  for  analyzing  the  effectiveness  of  the  JHMCS  human  factor 
issues.  Setting  an  average  favorable  rating  as  a  threshold  for  acceptance  will 
provide  the  necessary  insight  into  the  degree  of  satisfaction  acquired  from  its 
usage  and  the  additional  interviews  in  cases  of  ambiguous  results  and  the  use  of 
descriptive  analysis  will  provide  further  evidence  of  the  systems  operational 
suitability  in  relation  to  human  factors. 
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Recommendations,  Management  Conunents  and  Audit 
Response 


A.l.  We  recommend  that  the  Air  Force  Program  Executive  Officer  for 
Fighter  and  Bomber  Programs  update  the  Test  and  Evaluation  Master  Plan 
to  recognize  changes  as  a  result  of  the  program  restructure,  identify  the 
threat  threshold  and  enhancement  values  of  the  system,  and  establish  a 
process  to  update  on  a  regular  basis  the  Test  and  Evaluation  Master  Plan. 

Air  Force  Program  Executive  Officer  for  Fighter  and  Bomber  Programs 
Comments.  The  Air  Force  Program  Executive  Officer  concurred  and  stated 
that  the  Test  and  Evaluation  Master  Plan  was  being  updated  and  a  draft  was 
prepared  in  December  2000. 

A.2.  We  recommend  that  the  Commander,  Air  Force  Operational  Test  and 
Evaluation  Center,  revise  the  Multi-Service  Operational  Test  and^ 

Evaluation  Plan  to  include  changes  made  to  the  Test  and  Evaluation  Master 
Plan  that  affect  the  Plan,  identify  pass-and-fail  criteria  for  the 
questionnaires  used  to  evaluate  human  factors,  and  revise  the  statistical 
projection  plan. 

Air  Force  Program  Executive  Officer  for  Fighter  and  Bomber  Progr^s 
Comments.  The  Air  Force  Program  Executive  Officer  nonconcurred  with  the 
recommendation.  He  said  that  the  questionnaires  are  intended  to  gather 
subjective  information  to  better  understand  the  operational  impact  of  the  new 
system.  The  results  of  the  questioimaires,  along  with  other  data,  will  allow 
testers  to  render  an  opinion  on  operational  effectiveness  and  suitability. 

Air  Force  Operational  Test  and  Evaluation  Center  Comments.  The 
Commander,  Air  Force  Operational  Test  and  Evaluation  Center,  nonconcunred 
with  the  recommendation.  He  stated  that  the  Center  would  revise  the  Multi- 
Service  Operational  Test  and  Evaluation  Plan  to  include  changes  made  to  the 
Test  and  Evaluation  Master  Plan  as  updates  occur  but  would  not  agree  to  set 
pass-and-fail  criteria  for  the  human  factors  because  the  questionnaires  were 
designed  to  collect  subjective  information  in  assisting  the  user  to  understand  the 
effects  of  supporting  and  employing  the  system.  As  stated  in  his  comments  to 
the  finding,  he  mentioned  two  problems  that  we  identified  in  our  report 
pertaining  to  the  evaluation  of  human  factors.  The  problems  include  the  small 
nonrandom  sample  size  and  the  lack  of  quantifiable  pass-and-fail  criteria  for  the 
subjective  questionnaire.  He  stated  that  a  valid  statistical  projection  plan  would 
require  a  sample  that  exceeded  resource  availability  for  the  benefits  derived.  He 
stated  that  the  sample  the  Center  plans  to  use  was  sufficient  to  provide  clear 
answers  concerning  the  impact  of  human  factor  elements  on  the  operational 
effectiveness  and  suitability  because  the  test  pilots  employed  were  experts  with 
extensive  experience  in  determining  the  military  utility  of  a  system  under  test. 

In  addition,  for  subjective  critical  operational  issues,  measures  of  effectiveness, 
and  measures  of  performance,  the  Center  will  rate  the  system  as  favorable  if  the 
majority  of  the  ratings  are  positive  and  unfavorable  if  the  majority  of  the  ratings 
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are  unfavorable.  The  Center  will  also  conduct  additional  interviews  of  the  test 
pilots  where  the  ratings  do  not  provide  clear  answers.  In  addition,  the  Center 
will  use  descriptive  statistics  to  summarize  the  data  and  provide  a  narrative 
summary  of  the  strengths  and  weaknesses  of  the  system. 

Audit  Response.  The  plan  of  action  submitted  by  the  Director,  Air  Force 
Operational  Test  and  Evaluation  Center,  to  assess  the  systems'  operation^ 
suitability  in  relation  to  human  factors  met  the  intent  of  the  recommendation. 
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B.  Adequacy  of  Acquisition  Planning 

The  JHMCS  acquisition  approach  needed  improvement  because  the 
acquisition  plan  is  outdated,  did  not  address  low-rate  initial  production, 
did  not  recognize  the  risks  associated  with  the  restructure  and  the 
contracting  structure  of  the  JHMCS  program,  and  did  not  explore 
component  breakout  opportunities  for  full-rate  production.  As  a  result, 
the  evaluation,  identification,  and  management  of  contractor  performance 
is  at  risk.  In  addition,  the  joint  program  office  would  miss  the 
opportunity  to  put  funds  of  approximately  $17  million  to  better  use. 


DoD  Acquisition  Regulations 


The  DoD  Regulation  5000. 2-R,  “Mandatory  Procedures  for  Major  Defense 
Acquisition  Programs  and  Major  Automated  Information  System  Acquisition 
Programs,”  October  23,  2000,  provides  that  each  progrpi  manager  develop  an 
acquisition  strategy  to  minimize  the  time  and  cost  of  satisfying  a  need.  Essential 
elements  include  risk  management,  sources,  and  the  contract  and  management 
approach.  The  joint  program  office  must  develop  the  strategy  at  the  beginning 
of  a  program  and  update  it  when  there  is  a  change. 

The  Defense  Federal  Acquisition  Regulation  Supplement,  Subpart  207.1— 
Acquisition  Plans  October  25,  2000,  states  that  the  acquisition  plan  should  help 
officials  plan  for  the  evaluation,  identification,  and  management  of  contractor 
performance  risk.  In  addition,  it  should  include  a  milestone  chart  depicting  the 
acquisition  objectives.  The  acquisition  plan  can  embody  the  acquisition 
strategy. 

The  Defense  Federal  Acquisition  Regulation  Supplement  also  addresses 
component  breakout,  which  is  the  process  whereby  the  Government  purchases 
components  directly  from  the  manufacturer  and  furnishes  them  to  the  end-item 
manufacturer  as  Government-furnished  material.  This  process  eliminates  the 
prime  contractor  and  other  overheads  and  profits  and  achieves  savings  for  the 
Government.  It  is  DoD  policy  to  break  out  components  of  weapons  systems 
when  it  is  anticipated  through  a  breakout  analysis  that  a  prime  contract  will  be 
awarded  without  adequate  price  competition;  substantial  net  cost  savings 
probably  will  be  achieved;  and  the  quality,  reliability,  performance,  or  timely 
delivery  of  the  end  item  will  not  be  jeopardized. 

Single  Acquisition  Management  Plan 

The  Single  Acquisition  Management  Plan  (the  plan)  records  the  acquisition 
strategy  and  plan  for  the  JHMCS.  The  joint  program  office  updated  the  plan  in 
January  1997  to  support  the  Milestone  11  Engineering,  Manufacturing,  and 
Development  phase  decision.  The  plan  addressed  only  Milestone  II.  Because 
senior  Air  Force  officials  informed  the  joint  program  office  tlwt  a  revised  plan 
would  only  be  required  before  the  production  decision  in  April  2002,  the  joint 
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program  office  did  not  update  the  plan  to  recognize  risks  associated  with 
low-rate  initial  production,  the  program  restructure  approved  in  December 
1999,  the  contracting  structure  of  Ae  acquisition,  and  component  breakout. 

Low-Rate  Initial  Production.  The  acquisition  strategy  panel  (the  Panel) 
of  March  12,  1999,  addressed  low-rate  initial  production  for  the  F-18  E/F 
models.  The  Panel  reviewed  technical,  management,  and  cost  risks.  It  also 
rated  the  technical  risks  of  design  as  low  because  the  system  was  operating  on 
the  F-18  aircraft  with  only  minor  potential  for  redesign  work.  The  Panel  rated 
the  technical  risk  of  performance  as  moderate  because  of  ejection  safety 
qualifications  and  other  qualification  tests  in  progress.  The  Panel  rated 
management  risks  as  moderate  because  of  the  multi-Service  involvement  with 
many  platforms,  the  AIM-9X  missile  schedule  problems,  and  the  subcontractor 
management.  The  Panel  identified  the  cost  risk  as  moderate  because  of  the 
aggressive  schedule,  test  uncertainties,  AIM-9X  integration,  correction  of 
deficiencies,  and  reliability  of  components.  The  Panel  only  considered  cost 
when  it  evaluated  and  chose  a  firm-fixed-price  contract.  It  recommended 
modifying  the  strategy  to  incorporate  the  low-rate  initial  production  approach; 
however,  the  joint  program  office  did  not  revise  the  plan  or  reconvene  the  Panel 
before  low-rate  initial  production.  Further,  the  joint  program  office  did  not 
update  the  plan  to  recognize  the  recommendations  of  the  Panel  and  to  analyze 
the  technical  risks  involved. 

Program  Restructure.  The  program  restructure  of  December  1999 
changed  the  program’s  acquisition  strategy  by  adding  another  low-rate  initial 
production,  changing  the  schedule  for  production  until  April  2002,  and  adding 
funds  to  address  problems  with  the  maturity  of  the  High  Off  Boresight 
Capability  and  the  reliability  and  maintenance  of  the  helmet-vehicle  interface  for 
the  F-18  aircraft. 

The  joint  program  office  did  not  revise  the  plan  to  consider  how  the  risks  of 
program  restructure  affected  the  acquisition.  In  addition,  the  joint  program 
office  did  not  consider  the  risk  of  continuing  development  until  the  production 
decision  of  April  2002,  while  starting  low-rate  initial  production  on  the  F-18E/F 
aircraft  in  May  2000.  An  acquisition  strategy  and  plan  must  include  risk 
management,  the  management  approach,  and  a  milestone  chart  depicting  the 
acquisition  objective.  The  joint  program  office  did  not  update  the  acquisition 
strategy  and  plan  to  recognize  acquisition  changes  and  risks  resulting  from 
the  restructure. 

Structure  of  the  Acquisition.  A  hierarchical  structure  of  contractor 
tiers  exists  between  the  joint  program  office  and  the  manufacturers  of  the 
JHMCS.  The  Boeing  Company  is  the  prime  contractor  for  the  JHMCS  and  the 
integrator  for  the  F-18  and  F-15  aircraft.  Boeing  subcontracted  the  development 
effort  to  Vision  Systems  International,  a  hmited  liability  corporation  established 
by  two  firms,  Kaiser  Electronics  and  Elbit  Fort  Worth,  who  share  equally  in  the 
development  effort.  Kaiser  Electronics  and  Elbit  Fort  Worth  established  the 
structure  at  the  initiation  of  the  program  when  the  two  companies  formed  Vision 
Systems  International  to  share  in  the  development  and  risk.  Elbit  Fort  Worth 
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subcontracted  its  share  of  development  to  its  parent  organization  Elbit  Systems 
Limited,  in  Haifa,  Israel.  The  figure  below  shows  the  levels  of  contract 
management. 


Contractu^  Hierarchy  for  the  Joint  Helmet  Mounted  Cueing  System 

Vision  Systems  International  anticipated  that  the  sales  of  the  JHMCS  would 
provide  foil  production  at  all  three  facilities  --  Kaiser,  Elbit  Forth  Worth,  and 
Elbit  Israel.  Vision  Systems  International  officials  indicated  that  development 
and  low-rate  initial  production  was  occurring  only  at  Kaiser  Electronics  and 
Elbit  Systems  in  Israel.  The  other  contractor  levels  provide  some  m^keting  and 
development  support,  but  do  not  participate  in  the  actual  manufacturing  process. 

The  structure  of  contracting  for  the  JHMCS  includes  many  levels  of  overhead 
and  profit.  The  acquisition  plan  did  not  address  the  levels  of  contracting,  risk 
management,  sources,  contract  and  management  approaches,  and  benefits 
derived,  if  any,  by  the  Government  for  maintaining  the  current  contracting 
structure.  The  joint  program  office  did  not  revise  the  plan  after  the  restructure 
to  consider  these  factors. 

Breakout  Candidates.  The  acquisition  plan  did  not  address  component 
breakout  for  full-rate  production.  We  identified  five  potential  breakout 
components  for  the  JHMCS:  the  electronics  unit,  the  magnetic  tr^smitter  unit, 
the  cockpit  unit,  the  cathode  ray  tube,  and  the  helmet-mounted  display  test  set. 
We  reviewed  the  date  of  the  latest  configuration  change,  whefoer  the  component 
was  returned  for  rework,  the  reason  for  the  rework,  and  any  impact  on  the 
configuration.  See  the  table  below. 


14 


Component  Breakout  Candidates,  Date  of  Latest  Configuration  Change  and 

Returns  for  Rework 


Component  Description  Date  of  Latest  Returned  to  Vendor  for 

Configuration  Change  Rework  (Y/N) 


Electronics  Unit  05/04/99  Y 

Cathode  Ray  Tube  04/29/98  N 

Coclq)it  Unit  07/06/00  Y 

Magnetic  Transmitter  Unit  12/16/98  Y 

Test  Set  09/17/98  N 


Contractor  officials  at  Vision  Systems  International  and  Boeing  indicated  that  all 
the  components  we  identified  are  stable.  Two  components  were  not  returned  for 
rework.  Although  three  components  were  returned,  officials  stated  that  the 
rework  resulted  in  no  major  design  changes. 

The  DoD  Regulation  5000.2R  requires  consideration  of  component  breakout. 
The  acquisition  plan  should  address  the  risks  associated  with  breakout  such  as 
quality,  reliability,  performance,  and  timely  delivery.  As  the  program 
approaches  the  end  of  the  engineering,  manufacturing,  and  development  phase, 
the  joint  program  office  must  conduct  a  component  breakout  review  to  identify 
potential  candidates  for  breakout  during  full-rate  production. 

Benefits  of  Component  Breakout 

The  Air  Force  and  Navy  could  put  $16.9  million  to  better  use  if  the  joint 
program  office  purchased  the  five  components  firom  the  manufacturers  and 
delivered  them  to  Boeing  as  Government-furnished  equipment.  (Appendix  B. 
Estimated  Component  Breakout  Savings).  Our  calculation  excluded  overhead 
and  profit  for  two  management  levels,  Boeing  and  Vision  Systems  International. 
We  used  the  quantities  ftinded  from  2002  through  2005.  The  joint  program 
office  should  perform  component  breakout  reviews  for  all  components  as 
required  before  the  production  decision  and  include  the  results  in  the  acquisition 
plan  and  as  exit  criteria  for  the  current  acquisition  phase. 

Conclusion 

The  DoD  Regulation  5000.2R  provides  that  each  progrmi  nianager  develop  an 
acquisition  strategy  and  plan  to  include  the  evaluation,  identification,  and 
management  of  contractor  performance  risk,  contract  sources,  management 
approach,  and  milestone  chart  of  events.  The  joint  program  office  must  update 
the  acquisition  strategy  when  there  are  changes  that  affect  the  acquisition 
approach.  Also,  the  Regulation  requires  a  component  breakout  review  to  be 
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conducted  to  determine  whether  substantial  net  cost  savings  can  be  achieved 
through  breakout  without  jeopardizing  the  quality,  reliability,  performance,  or 
timely  delivery  of  the  end  item.  The  JHMCS  joint  program  manager  did  not 
update  the  acquisition  plan  to  recognize  recommendations  of  the  acquisition 
strategy  panel  for  the  low-rate  initial  production,  the  impact  of  the  program 
restructure,  the  contracting  structure  of  the  JHMCS  acquisition,  md  the 
potential  savings  generated  by  component  breakout  once  production  begins. 
The  acquisition  plan  needed  to  be  updated  to  consider  the  changing  facets  of 
the  program. 

Management  Comments  on  the  Finding  and  Audit  Response 


Management  Comments  on  Acquisition  Planning.  The  Air  Force  Program 
Executive  Officer  nonconcurred  with  the  finding  that  acquisition  planning  is 
outdated.  He  stated  that  although  the  Single  Acquisition  Management  Plan  has 
not  been  updated  since  approved  in  1996,  the  planning  and  strategy  have  been 
updated  in  steering  group  and  strategy  panel  meetings,  which  the  milestone 
authority  approved.  Those  activities  occurred  in  the  context  of  an  acquisition 
plan  update.  He  agreed  that  the  Single  Acquisition  Management  Plan  will  be 
updated  to  consolidate  program  changes  and  documentation  since  the  last 
milestone  review. 

The  Air  Force  Program  Executive  Officer  also  nonconcurred  that  the  risk 
assessment  process  was  inadequate.  He  stated  that  they  followed  the 
Aeronautical  Systems  Center  policies  and  procedures  and  conducted  risk 
analysis  for  low  rate  initial  production  1  and  2.  In  addition,  they  have  extensive 
risk  assessment  processes  for  all  technical  issues  of  the  program.  He  stated  that 
they  manage  technical,  programmatic,  and  integration  risk  across  the  program  in 
a  prudent  and  cost-effective  manner. 

The  Program  Executive  Officer  further  commented  that  component  breakout  is 
not  considered  possible  until  development  is  completed.  He  nonconcurred  with 
our  position  that  the  JHMCS  acquisition  plan  does  not  address  potential  cost 
savings  of  component  breakout  for  full  rate  production.  He  also  nonconcurred 
with  the  estimated  savings  of  $17  million  and  stated  our  analysis  is  flawed 
because  we  eliminated  the  Boeing  profit  and  the  VSI  profit,  which  is  in  conflict 
with  the  DoD  acquisition  regulations.  He  suggested  the  savings  would  be  about 
$5  million.  The  Program  Executive  Officer  also  made  additional  comments  on 
the  report’s  background  and  appendix  sections.  Additional  comments  are 
included  in  the  Appendix  A. 

Audit  Response.  We  recognize  that  the  program  has  steering  group  and 
strategy  panel  meetings  to  address  low  rate  initial  production  and  technical 
challenges,  but  the  results  were  not  included  in  the  acquisition  plan.  By  doing 
so,  the  plan  would  capsulate  the  acquisition  approach  and  strategy  of  the 
JHMCS  in  one  document  instead  of  several.  Revising  the  acquisition  plan 
would  also  embody  the  results  of  changes  in  strategy  and  ensure  that  the  Air 
Force  and  other  interested  parties  would  have  the  current  approach  to 
acquisition  in  one  document.  In  addition,  the  DoD  Regulation  5000. 2-R, 
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requires  program  managers  to  develop  an  acquisition  strategy  and  plan  to 
address  risk  management,  sources,  and  the  contract  and  management  approach, 
and  to  update  them  when  there  is  a  change.  This  was  not  done. 

Concerning  risk  assessments,  our  report  did  not  state  that  the  risk  assessment 
process  was  inadequate.  Rather,  it  stated  that  the  acquisition  approach  was 
inadequate  because  the  acquisition  plan  did  not  address  low-rate  initial 
production,  did  not  recognize  the  risks  associated  with  the  restructure  and  the 
contracting  structure  of  die  JHMCS  program,  and  did  not  explore  component 
breakout  opportunities  for  full-rate  production.  The  report  indicated  that  the 
acquisition  panel  for  the  low-rate  initial  production  recommended  modifying  the 
strategy  to  incorporate  the  low-rate  initial  production  approach;  however,  the 
joint  program  office  did  not  revise  the  plan  or  reconvene  the  Panel  before  low- 
rate  initial  production.  In  addition,  the  joint  program  office  did  not  update  the 
plan  to  recognize  the  recommendations  of  the  Panel  and  to  analyze  the  technical 
risks  involved. 

Concerning  the  restructure,  the  acquisition  plan  did  not  consider  the  risfe  of 
continuing  development  until  the  production  decision  of  April  2002,  while  low- 
rate  initial  production  started  on  Ae  F-18E/F  aircraft  in  May  2000.  An 
acquisition  strategy  and  plan  must  include  risk  management,  the  management 
approach,  and  a  milestone  chart  depicting  the  acquisition  objective.  The  joint 
program  office  did  not  update  the  acquisition  strategy  and  plan  to  recognize 
acquisition  changes  and  risks  resulting  from  the  restructure. 

The  Program  Executive  Officer  apparently  did  not  understand  that  the  Boeing 
and  VSI  profits  would  be  excluded  on  a  component  breakout  basis  because 
Boeing  and  VSI  would  not  be  involved  in  the  acquisition.  Our  $17  million 
calculation  excludes  the  Boeing  and  VSI  profits  ^cause  the  components  would 
be  purchased  directly  from  the  manufacturers  Kaiser  and  Elbit,  Fort  Wo^. 

The  components  would  then  be  shipped  to  Boeing  or  Lockheed  and  provided  as 
Government-furnished  supplies.  The  $17  million  is  a  target  amount  and  we 
understand  that  actual  contract  negotiations  will  result  in  a  different  amount.  As 
such,  we  will  perform  followup  to  determine  the  amount  of  monetary  benefits 
the  Air  Force  will  actually  achieve.  Concerning  the  comments  on  the 
background  and  executive  summary  sections,  we  made  appropriate  changes. 
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Recommendations,  Management  Comments,  and  Audit 
Response 


B.  We  recommend  that  the  Air  Force  Program  Executive  Officer  for  ^ 
Fighter  and  Bomber  Programs  require  the  Joint  Helmet  Mounted  Cueing 
System  program  manager  to: 

1.  Conduct  and  present  a  component  breakout  study  as  esdt  criteria 
for  the  engineering,  manufacturing,  and  development  phase  that  includes 
an  analysis  of  potential  net  cost  savings  that  can  be  achieved;  and  a  review 
of  the  quality,  reliability,  performance,  and  the  timely  delivery  of  the  end 
item  that  may  be  jeopardized. 

Management  Comments.  The  Air  Force  Program  Executive  Officer 
concurred,  and  stated  that  his  office  provided  direction  to  the  program  office  to 
conduct  a  breakout  study  in  August  2000. 

2.  Revise  the  acquisition  plan  to  include  the  results  of  the  breakout 
study,  the  risks  associated  with  low-rate  initial  production  while  in 
development,  the  restructure  of  the  program,  and  the  contracting  structure 
of  acquisition. 

Management  Comments.  The  Air  Force  Program  Executive  Officer 
concurred,  and  stated  that  the  acquisition  plan  will  be  updated  to  support 
Milestone  3. 

3.  Elstablish  a  process  to  update  the  acquisition  plan  on  a  regular 

basis. 

Management  Comments.  The  Air  Force  Program  Executive  Officer 
nonconcurred,  stating  that  no  special  process  is  required  because  the  acquisition 
planning  document  will  continue  to  be  updated  consistent  with  DoD  and  Air 
Force  policy. 

Audit  Response.  The  Air  Force  comments  are  not  responsive.  We  do  not 
believe  that  the  JHMCS  program  office  has  a  process  because  if  it  did  the 
acquisition  plan  would  have  been  updated  at  the  time  of  the  rebaseline  or  other 
critical  events  as  mandated  by  DoD  Regulation.  Therefore,  we  request  that  the 
Air  Force  provide  additional  comments  in  response  to  the  final  report. 
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Appendix  A.  Audit  Process 

Scope  and  Methodology 


The  overall  audit  objective  was  to  evaluate  the  acquisition  of  the  JHMCS. 
Specifically,  the  au^t  determined  whether  the  Air  Force  is  cost-effectively 
readying  the  system  for  the  production  phase  of  the  acquisition  process,  '^e 
audit  was  performed  in  accordance  with  the  Inspector  General,  DoD,  critical 
program  management  element  approach,  and  we  reviewed  program  management 
elements  pertaining  to  requirement  evolution  and  affordability,  test  and 
evaluation,  contracting,  and  joint  management.  We  reviewed  program  data 
from  December  1996  through  November  2000.  We  also  evaluated  the 
management  control  program  as  it  related  to  the  overall  objective. 

We  performed  this  economy  and  efficiency  audit  from  June  2000  through 
November  2000  according  to  standards  implemented  by  the  Comptroller 
General  for  the  United  States,  as  implemented  by  the  Inspector  General,  DoD. 
We  used  criteria  in  the  DoD  Regulation  5000. 2R  to  perform  the  audit.  To 
accomplish  the  audit  objectives,  we  determined  that  the  JHMCS  joint  program 
management  office  had  developed  and  implemented  an  acquisition  plan  and  a 
test  and  evaluation  plan. 

Use  of  Computer-Processed  Data.  We  did  not  use  computer-processed  data  to 
perform  this  audit.  The  Technical  Assessment  Division,  Audit  Followup  and 
Technical  Support  Directorate,  Office  of  the  Inspector  General,  provided 
expertise  in  the  area  of  testing  including  operational  test  planning. 

Contacts  During  the  Audit.  We  visited  or  contacted  individuals  and 
organizations  within  the  Departments  of  the  Air  Force  and  the  Navy.  We  also 
visited  or  contacted  individuals  and  organizations  within  DoD  and  contractor 
and  subcontractor  officials. 

DoD-Wide  Corporate  Level  Government  Performance  and  Results  Act 
Coverage.  In  response  to  the  Government  Performance  and  Results  Act,  the 
Secretary  of  Defense  annually  establishes  DoD-wide  corporate  level  goals, 
subordinate  performance  goals,  and  performance  measures.  This  report  pertains 
to  achievement  of  the  following  goal  and  subordinate  performance  goal. 

FY  2001  DoD  Corporate  Level  Goal  2:  Prepare  now  for  an  uncertain  future 
by  pursuing  a  focused  modernization  effort  that  maintains  U.S.  qualitative 
superiority  in  key  warfighting  capabilities.  Transform  the  force  by  exploiting 
the  Revolution  in  Military  Affairs,  and  reengineer  the  Department  to  achieve  a 
21st  century  infrastructure.  (Ol-DoD-02). 

FY  2001  Subordinate  Performance  Goal  2.4:  Meet  combat  forces’  needs 
smarter  and  faster,  and  products  and  services  that  work  better  and  cost  less,  by 
improving  the  efficiency  of  DoD  acquisition  processes.  (Ol-DoD-2.4) 
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General  Accounting  Office  High-Risk  Area.  The  General  Accounting  Office 
had  identified  several  high-risk  areas  in  the  DoD.  This  report  provides 
coverage  of  the  Defense  Weapon  System  Acquisition  high-risk  area. 

Management  Control  Program  Review 

DoD  Directive  5010.38,  “  Management  Control  (MC)  Program,”  August  26, 
1996,  and  DoD  Instruction  5010.40,  “Management  Control  (MC)  Program 
Procedures,”  require  DoD  managers  to  implement  a  comprehensive  system  of 
management  controls  that  provide  reasonable  assurance  that  programs  are 
operating  as  intended  and  to  evaluate  the  adequacy  of  the  controls. 

Scope  of  the  Review  of  the  Management  Control  Program.  In  accordance 
with  DoD  Directive  50(X).l,  “Defense  Acquisition  System,”  October  23,  2000, 
and  Department  of  Defense  Regulation  5000.2-R,  “Mandatory  Procedures  for 
Major  Defense  Acquisition  Programs  and  Major  Automated  Information  System 
Acquisition  Programs,”  dated  October  23,  2000,  acquisition  managers  are  to  use 
program  cost,  schedule,  and  performance  parameters  as  control  objectives  to 
implement  the  requirements  of  DoD  5010.38.  Accordingly,  we  limited  our 
review  to  management  controls  directly  related  to  the  acquisition  management  of 
the  JHMCS. 

Adequacy  of  Management  Controls.  In  evaluating  the  management  control 
process,  we  identified  material  management  control  weaknesses  for  the  JHMCS 
joint  program  office  as  defined  by  DoD  Instruction  5010.40.  The  JHMCS  joint 
program  office  management  controls  for  updating  the  acquisition  and  testing 
plans  were  not  adequate  to  ensure  that  the  plans  contained  current  information 
on  the  status  and  risks  associated  with  the  program.  Recommendations  A.  1.  and 
B.  2.,  if  implemented,  will  improve  the  updating  process  Md  procedures,  and 
could  result  in  potential  monetary  benefits  of  about  $17  million  (Appendix  B). 

A  copy  of  the  report  will  be  provided  to  the  senior  officials  responsible  for 
management  controls  m  the  Air  Force. 

Adequacy  of  Management’s  Self-Evaluation.  In  evaluating  the  management 
control  process,  we  reviewed  the  risk-management  program  to  determine  the 
level  of  risk  that  the  officials  assigned  to  aspects  of  the  helmet.  We  also 
reviewed  the  Annual  Statements  of  Assurance  for  the  Air  Force  for  FYs  1998 
and  1999  to  determine  whether  any  weaknesses  had  been  reported  relating  to  the 
JHMCS  program.  Air  Force  officials  did  not  identify  procurement  and  testing 
of  the  helmet  as  an  assessable  unit  and  therefore  did  not  identify  or  report  the 
material  management  control  weaknesses  identified  by  the  audit. 


Prior  Coverage 


During  the  last  5  years,  there  has  been  no  prior  coverage  on  the  JHMCS. 
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Management  Comments  on  Management  Control  Weaknesses 


Management  Comments.  The  Air  Force  Program  Executive  Officer  stated  that 
test  plans  are  living  documents  and  are  constantly  being  updated.  In  addition, 
the  acquisition  strategy  and  risk  assessments  are  clearly  documented  in  other 
than  the  Single  Acquisition  Management  Plan.  Accordingly,  the  controls  for 
managing  the  testing  and  acquisition  plamiing  are  adequate. 

Audit  Response.  While  plans  are  in  place  to  address  testing  and  acquisition, 
they  are  not  current  and  do  not  portray  the  status  of  the  program  after  the 
rebaseline  and  other  programmatic  changes.  Since  there  was  no  process  in 
place  to  update  the  testing  and  acquisition  documents,  this  represents  a 
management  control  weataiess. 
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Appendix  B.  Component  Breakout 
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^  Contractor  proprietary  data  removed. 


Appendix  C.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  (Comptroller) 

Deputy  Chief  Financial  Officer 
Deputy  Comptroller  (Program/Budget) 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
Director,  Operational  Test  and  Evaluation 


Department  of  the  Navy 

Naval  Inspector  General 

Auditor  General,  Department  of  the  Navy 


Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 

Auditor  General,  Department  of  the  Air  Force 

Air  Force  Program  Executive  Officer,  Fighter  and  Bomber  Programs 


Non-Defense  Federal  Organization 

Office  of  Management  and  Budget 

Congressional  Committees  and  Subcommittees,  Chairman  and 
Ranking  Minority  Member 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Coimnittee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Armed  Services 
House  Committee  on  Government  Reform 

House  Subcommittee  on  Government  Efficiency,  Financial  Management,  and 
Intergovernmental  Relations,  Committee  on  Government  Reform 
House  Subcommittee  on  National  Security,  Veterans  Affairs,  and  International 
Relations,  Committee  on  Government  Reform 
House  Subcommittee  on  Technology  and  Procurement  Policy,  Committee  on 
Government  Reform 
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Department  of  the  Air  Force  Comments 


DEPARTMENT  OF  THE  AIR  FORCE 
AIR  FORCE  PROGRAM  EXECUTIVE  OFFICE 
WASHINGTON.  DC  20330-1060 

3  0  JAN  2001 


MEMORANDUM  FOR^  ASSISTANT  INSPECTOR  GENERAL  FOR  AUDITING 
OFFICE  OF  THE  INSPECTOR  GENERAL 
DEPARTMENT  OF  DEF  ENSE 

FROM:  AFPEO/FB 

1060  Air  Force  Pentagon 
Washington  DC  20330-1 060 

SUBJECT;  Audit  Report  on  the  Acquisition  of  the  Joint  Helmet  Mounted  Cueing  System 
(Project  No.  D2000AB-0220),  dated  December  8, 2000 

This  is  in  reply  to  your  memorandum  requesting  the  Assistant  Secretary  of  the  Air  Force 
(Financial  Management  and  Comptroller)  provide  Air  Force  comments  on  the  subject  report 
Specifically,  you  requested  the  Air  Force  Program  Executive  Officer  for  Fighter  and  Bomber 
Programs  (AFPEO/FB)  and  the  Director  (sic),  Air  Force  Operational  Test  and  Evaluation  Center 
(AFOTEC)  to  comment  on  this  report  AFOTEC  will  provide  separate  comments.  The 
comments  in  this  document  have  been  coordinated  with  the  JHMCS  Joint  Program  Office 
(JPO),  Air  Combat  Command,  HQ  USAF,  DOT&E,  PEO(T).  and  NAVAIR  staffs. 

Air  Force,  Navy,  and  Contractor  JHMCS  personnel  spent  more  than  400  man-hours 
cfiscussing  JHMCS  program  details  and  providing  JHMCS  program  documentation  to  the  DoD 
IG  Team.  The  subject  report  contradicts  Information  provided  to  the  IG  both  verbally  and  via 
program  documentation  concerning  several  of  the  report  findings  and  recommendations. 
Specific  examples  of  such  contradictions  are  included  in  the  attachment. 

Citing  potential  for  release  of  proprietary  Information,  the  IG  denied  the  JHMCS  System 
Program  Director  initial  recommendation  to  seek  comment  by  JHMCS  contractors.  However, 
the  JHMCS  program  office  has  reviewed  the  report  and  affirms  that  the  report  can  be  provided 
to  Boeing,  the  JHMCS  prime  contractor,  based  upon  current  security  and  policy  review 
procedures.  Information  from  the  prime  contractor  can  be  very  helpful  to  guarantee  the  IG 
retort  ultimately  reflects  an  accurate  representation  of  the  JHMCS  program. 

My  point  of  contact  for  this  project  is  Maj  Aaron  Claric.  703-588-7314  or 
aaron.clafk@pentaoon.af.mil. 

MICHAEL  C.  MUSHALA.  Maj  Gen.  USAF 
Air  Force  Program  Executive  Officer 
for  Fighter  and  Bomber  Programs 


Atch:  Detailed  comments  to  Audit  Report  on  the  Acquisition  of  the  Joint  Helmet 

Mounted  Cueing  System  (Project  No.  D2000AB-0220),  dated  December  8.  2000 


Detailed  Comments  to  Audit  Report  on  the  Acquisition  of  the  Joint 
Helmet  Mounted  Cueing  System  (Project  No.  D2000AB-0220), 
Dated  December  8, 2000 


DoD  IG  Finding  A.  UtHltv  of  Planned  Operation  Testing: 

1.  “The  JHMCS  operational  tests  will  not  provide  objective  test  results  necessary  to 
support  the  JHMCS  full-rate  production  decision  In  April  2002  because  of  the 
following: 

•  The  Operational  Requirements  Document  (ORD)  did  not  identify  operational 
parameters  and  articulate  requirements  in  measurable  terms 

Response;  Non-Concur 

Discussion:  As  a  cueing  and  display  system,  the  ORD  requires  JHMCS  to  cue  the  radar, 
navigation  system,  current  and  next  generation  short-range  missiles,  and  to  display  on  the  visor 
information  from  these  systems  in  a  usable  format  similar  to  the  aircraft  head-up  display  (HUD). 
Further,  the  ORD  requires  JHMCS  to  cue  these  systems  from  -1G  to  7.2G,  to  provide  a  means 
to  cue  beyond  the  physical  limitations  of  the  neck  (uplooks),  and  to  display  the  current 
line-of-sight  (LOS)  of  each  system.  Objective  technical  requirements  for  display  symbology, 
brightness  levels,  uplook  limits,  and  cueing  accuracy  are  contained  In  other  technical  and 
engineering  documents.  If  JHMCS  satisfies  the  above  key  performance  pararneters  with  both 
current  and  future  vi^apons  (e.g.,  A1M-9X),  it  follows  that  JHMCS  will  enable  pilots  to  employ 
weapons  more  quickly  and  achieve  the  first-shot  advantage  described  in  the  ORD.  As  stated  in 
the  ORD  and  other  program  documents.  HMCS  is  a  cueing  and  information  display  device  that 
will  benefit  the  warfighter  by  allowmg  aircrew  to  fully  exploit  off-boresight  capabilities  of  fighter 
radars  and  missiles  and  to  display  vital  weapons  and  target  information  for  target  verification 
and  situational  awareness.  Finally,  the  ORD  identifies  as  specific  threat  examples  the 
Russian’s  MiG-29  and  Su-27  fighters  equipped  with  a  helmet-mounted  sight  and  advanced 
AA-11  missile.  The  capabilities  of  these  aircraft  were  used  to  establish  objective  evaluation 
criteria  In  the  classified  TEMP  annex  that  will  be  used  during  test.  Failure  of  the  system  to  cue 
weapons/sensors  and  display  HUD  information  will  be  documented  during  test  and  will  be 
reflected  in  objective,  measurable  Readiness  and  Logistics  ratings  for  operational  suitability  and 
effectiveness. 


DOD  16  Finding  A.  Utilitv  of  Planned  Operation  Testing: 

2.  “The  JHMCS  operational  tests  will  not  provide  objective  test  results  necessary  to 
support  the  JHMCS  full-rate  production  decision  in  April  2002  because  of  the 
following: 

•  The  Test  and  Evaluation  Master  Plan  (TEMP)  Is  outdated  and  insufficient  to 
provide  the  overall  structure  for  an  objective  testing  program  and  to  ensure  that 
the  operational  tests  will  provide  objective  results  that  can  determine  whether  the 
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program  Is  operationally  effective  and  suitable  for  meeting  the  warfighters*  needs 
in  entering  production.” 

Response:  Concur 

Discussion:  The  TEMP  as  currently  written  is  outdated  and  is  being  updated.  In  Dec  00,  the 
JHMCS  Joint  Program  Office  released  a  draft  update  that  is  under  review  by  the  JHMCS 
Integrated  Product  Team  (IPT). 


DoD  IG  Finding  A.  Utility  of  Planned  Operation  Testing: 

3.  “The  JHMCS  operational  tests  will  not  provide  objective  test  results  necessary  to 
support  the  JHMCS  full-rate  production  decision  in  April  2002  because  of  the 
following: 

•  The  Multi-Service  Operaitionat  Test  and  Evaluation  Plan  did  not  include  a  baseline 
threshold  or  objectives  to  measure  success,  did  not  include  plans  for  a  valid 
statistical  projection,  did  not  specify  pass-and-fai!  criteria,  and  did  not  include  a 
confidence  level  for  questionnaires  developed  to  measure  and  project  human 
factor  elements  of  critical  operational  issues.” 

Response:  Non-Concur 

Discussion:  The  Multi-Service  Operational  Test  and  Evaluation  Plan  is  consistent  with  the 
original  TEMP  and  contains  numerous  test  procedures  to  satisfy  Critical  Operational  Issues 
(COIs)  and  meet  identified  Measures  of  Perfomiance  (MOPs).  The  updated  Multi-Senrlce 
Operational  Test  and  Evaluation  Plan  will  be  consistent  with  the  updat^  TEMP,  and  will  include 
test  procedures  to  evaluate  JHMCS  capabilities  against  the  example  MiG-29  and  Su-27  threat 
aircraft  with  AA-11  missile  and  helmet-mounted  sight  Results  will  be  factored  into  the 
operational  effectiveness  assessment 

Responses  to  the  human  factors  questionnaires  will  depend  on  the  judgment  of  the  testers  to 
make  professior^al  assessments  on  human  factor  implications  associated  with  JHMCS  operation 
and  employment  Regardless  of  statistical  analysis  techniques  and  sampling  methods,  these 
questionnaires  will  give  the  using  commands  Insight  Into  system  characteristics  that  can  be 
used  to  develop  training,  to  refine  operational  concepts  of  operation  and  maintenance,  and  to 
suggest  fbllow-on  improvements. 


DoD  IG  Recommendation  (A.I.); 

“We  recommend  that  the  Air  Force  Program  Executive  Officer  for  Fighters  and  Bombers 
(sic)  update  the  Test  and  Evaluation  Master  Plan  to  recognize  changes  as  a  result  of  the 
program  restructure,  identify  the  threat  threshold  and  enhancement  values  of  the 
system,  and  establish  a  process  to  update  on  a  regular  basis  the  Test  and  Evaluation 
Master  Plan.” 

Response:  Concur 
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•  The  TEMP  will  be  updated.  In  Dec  00,  the  JHMCS  Joint  Program  Office  released  a  draft 
update  that  is  under  review  by  the  JHMCS  IPT. 

•  Although  DoD  5000.2-R  (Interim)  is  open  to  interpretation  as  to  whether  a  threat  baseline 
must  be  specified  in  the  TEMP,  particularly  for  ACAT  ill  programs,  a  description  of  the  threat 
HOBS  capability  will  be  added  to  the  classified  appendix  for  the  TEMP  update. 

•  The  TEMP  vwll  be  updated  in  accordance  with  processes  specified  In  AFI  99-102. 


DoD  IG  Recommendation  fA.2.); 

recommend  that  the  Director  (sic),  Air  Force  Operational  Test  and  Evaluation  Force 
(sic),  revise  the  Multi-Service  Operational  Test  and  Evaluation  Plan  to  include  changes 
made  to  the  Test  and  Evaluation  Master  Plan  that  affect  the  Plan,  identify  pass  and  fail 
criteria  for  the  questionnaires  used  to  evaluate  human  factors,  and  revise  the  statistical 
projection  plan.” 

Response:  Non-concur 

•  The  OT  Plan  will  be  revised  to  reflect  the  TEMP  currently  under  revision.  However,  we  do 
not  concur  with  the  recommendation  to  identify  pass/fail  criteria  for  the  human  factors 
questionnaires.  The  questionnaires  are  intended  to  gather  subjective  inputs  and  will  help 
the  user  better  understand  the  operational  impacts  of  supporting  and  employing  this  new 
capability.  As  stated  in  the  Test  Plan,  questionnaire  results  will  be  supplemented  with 
additional  information,  such  as  test  team  observations,  interviews,  mission  debriefs,  and 
deficiency  reports.  It  will  be  this  volume  of  data  that  will  allow  the  testers  to  render  their 
professional  opinion  on  operational  effectiveness  and  suitability. 

•  Further  comment  is  deferred  to  AFOTEC. 


DOD  IG  Finding  B,  Adequacy  of  Acquisition  Planning 

"The  JHMCS  acquisition  approach  is  not  effective  because  the  acquisition  plan  is 
outdated,  does  not  address  low  rate  initial  production,  dobs  not  recognize  the  risks 
associated  with  the  restructure  and  the  contracting  structure  of  the  JHMCS  program,  and 
does  not  explore  component  breakout  opportunities  for  full-rate  production.  As  a  result 
the  Joint  program  office  will  miss  the  opportunity  to  put  funds  of  approximately  $17 
minion  to  better  use.” 

Response:  Non-Concur 

Discussion:  Non-concur  with  IG  that  JHMCS  Acquisition  Planning  is  outdated.  JHMCS 
Acquisition  Planning  Is  current  and  documented.  DoD  5000.2-R  requires  an  Acquisition 
Strategy  be  developed  and  updated:  "The  PM  shall  Initially  develop  the  acquisition  strategy  at 
program  initiation  and  shall  keep  the  strategy  current  by  updating  it  whenever  there  is  a  change 
to  an  approved  strategy..."  AFFARS  5307.103.9  requires  System  Acquisition  Management 
Plans  (SAMP)  only  for  ACAT  I  and  11  programs.  JHMCS  Is  an  ACAT  111  program  and  as  such  is 
not  required  to  have  a  formal  SAMP.  However,  due  to  development  complexity  and  as  an 
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acquisition  initiative  a  JHMCS  SAMP  was  developed  and  coordinated  by  each  service  at 
Milestone  II  (start  of  engineering  and  manufacturing  development).  While  the  actual  JHMCS 
SAMP  document  has  not  been  updated  since  initial  approval  in  Nov  96,  the  acquisition  planning 
and  strategy  have  been  updated  several  times.  These  include: 

•  LRIP1  Acquisition  Strategy  Pane!  (ASP)  Briefing  and  Minutes— March  99 

•  HOBS  General  Officer  Steering  Group  Briefing  and  Minutes— December  99 

•  JHMCS  Acquisition  Program  Baseline  (APB)  update— March  00 

•  LRIP1  Acquisition  Review  Board  (ARB)  Briefing,  Minutes  and  Direction— May  00 

•  LRIP2  Acquisition  Strategy  Panel  Briefing  and  Minutes— Aug  00 

All  of  these  activities  were  reviewed  and  concurred  with,  in  writing,  by  the  Milestone  Decision 
Authorify,  the  Air  Force  Program  Executive  Officer  for  Fighter  and  Bomber  Programs  wth 
representation  by  the  Navy  Program  Executive  Officer  (Tactical).  These  activities  occurred  in 
the  context  of  an  acquisition  planning  update  and  were  so  noted  in  the  minutes.  The  JHMCS 
acquisition  planning  strategy  Is  not  outdated.  The  acquisition  strategy  is  current  and 
documented.  The  SAMP,  TEMP  and  Joint  Sen/ice  MOA  will  be  updated  prior  to  Milestone  3 
(full  rate  production)  to  fully  consolidate  program  changes  and  documentation  since  the  last 
milestone  review. 

Non-concur  with  I6  that  JHMCS  risk  assessment  processes  are  inadequate.  JHMCS  risk 
assessment  processes  are  well  established.  The  JHMCS  program  follows  Aeronautical  System 
Center  (ASC)  policies  and  procedures  for  conducting  Acquisition  Strategy  Panel  assessments 
prior  to  new  contract  awards.  The  ASC  process  involves  aggressive  risk  analysis  and 
evaluation  of  alternatives  In  all  functional  areas.  This  aggressive  risk  assessment  was 
completed  for  both  LRIP1  and  LRIP2.  The  Navy  also  required  extensive  risk  analysis  as  part  of 
their  LRIP1  ARB  process  and  was  so  documented. 

The  JHMCS  program  and  contractor  have  an  extensive,  monthly,  on-going  risk  assessment 
process  for  all  technical  program  issues.  It  is  this  risk  program  that  Identified  significant  HOBS 
integration  Issues  with  the  AIM-9X  on  the  F-15C/D  aircraa  The  JHMCS  risk  process  also 
identified  potential  operability  and  supportabillty  issues  in  advance  of  the  F/A-18  operational 
assessment  completion.  The  December  1999  JHMCS  program  restructure  was  accomplished 
in  order  to  reduce  overall  program  technical  risk.  The  insertion  of  LRIP1  and  LRiP2  was 
accomplished  after  careful  consideration  of  diverse  aircraft  platform  integration,  retrofit  and 
forward  fit  installation  requirements,  and  risks.  For  example,  the  LRIP2  date  was  specifically 
established  to  allow  combined  DT/OT  testing  of  redesigned  components  prior  to  a  production 
decision.  The  JHMCS  restructure  manages  technical,  programmatic  and  Integration  risk  across 
four  aircraft  types  in  a  prudent  and  cost  effective  manner. 

Non-concur  with  IG  that  JHMCS  Acquisition  Planning  does  not  address  the  potential  cost 
savings  of  component  breakout  for  full  rate  production.  JHMCS  considered  component 
breakout  as  part  of  both  the  LRIP1  and  LRIP2  ASP  process.  Component  breakout  was  viewed 
as  not  acceptable  due  to  orvgoing  devetopment  risks,  the  immaturity  of  toe  principal 
subcontractor  and  the  inability  of  the  team  to  quantify  cost  benefits.  At  toe  LRIP2  ASP  the 
JHMCS  Program  Director  recommended,  and  toe  PEO  concurred,  that  a  breakout  analysis  be 
completed  prior  to  Milestone  3.  This  analysis  will  be  initiated  In  toe  spring  of  2001 . 

Non-concur  wHh  IG  estimated  savings  of  $17  million.  The  )G  analysis  is  flawed. 

Referenced  analysis  assumes  that  all  principal  participants  In  toe  program — Boeing,  VSl,  Elbit 
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Fort  Worth,  Eibit  Haifa,  and  Kaiser  Electronics,  add  profit.  The  auditor  analysis  then  eliminates 
the  Boeing  proftt  ($5  million)  and  the  VSl  profit  ($1 1 .8  million).  This  analysis  provides  the 
contractor  team  with  zero  profit  since  the  VSl  business  arrangement  with  Kaiser,  Eibit  Fort 
Worth  and  Eibit  Haifa  allows  for  only  VSl  charging  profit.  Zero  profit  is  in  conflict  with  DoD 
acquisition  regulations  which  direct  reasonable  profit  be  paid  to  contractors.  Reinstatement  of 
the  VSl  profit  reduces  the  IG  estimated  savings  to  $5  million  from  $17  million.  While  $5  Million 
in  potential  savings  Is  significant,  it  Is  still  less  than  1%  of  the  planned  production  buy.  These 
savings  will  require  validation  relative  to  increased  cost  and  risks  associated  with  government 
vendor  management,  GFE  risk,  and  Integration  risk. 


DoD  IG  Recommendation  fB): 

'We  recommend  that  the  Air  Force  Program  Executive  Officer  for  Fighters  and  Bombers 

(sic)  require  the  Joint  Helmet  Mounted  Cueing  System  program  manager  to: 

1.  Conduct  and  present  a  component  breakout  study  as  exit  criteria  for  the 
Engineering,  Manufacturing,  and  Development  phase  (sic)  that  includes  an 
analysis  of  potential  net  cost  savings  that  can  be  achieved;  and  a  review  of  the 
quality,  reliability,  performance,  and  the  timely  delivery  of  the  end  item  that  may 
be  jeopardized” 

Response:  Concur 

•  AFPEO/FB  provided  this  direction  to  the  HMCS  SPD  at  the  LRIP2  ASP  in  Aug  2000. 


DoD  iG  Recommendab'on  fB): 

“We  recommend  that  the  Air  Force  Program  Executive  Officer  for  Fighters  and  Bombers 
(sic)  require  the  Joint  Helmet  Mounted  Cueing  System  program  manager  to: 

2.  Revise  the  acquisition  plan  to  include  the  results  of  the  breakout  study,  the  risks 
associated  with  low-rate  initial  production  while  in  development,  the  restructure  of 
the  program,  and  the  contracting  structure  of  acquisition.” 

Response:  Concur 

•  The  Single  Acquisition  Management  Plan  will  be  updated  to  support  Milestone  3. 


DoD  IG  Recommendation  (B.3.): 

“We  recommend  that  the  Air  Force  Program  Executive  Officer  for  Fighters  and  Bombers 
(sic)  require  the  Joint  Helmet  Mounted  Cueing  System  program  manager  to: 

3.  Establish  a  process  to  update  the  acquisition  plan  on  a  regular  basis.” 

Response:  Non-Concur 
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Final  Report 
Reference 


No  Special  process  is  required.  Acquisition  planning  documentation  will  continue  to  be  updated 
consistent  with  DoD  5000  and  the  ASC  ASP  process. 


Pop  IG  Report  Appendix  A.  Audit  Process.  Management  Control  Program  Review, 
Adequacy  of  Management  Controls 

Non-concur  with  the  IG’s  assertions  that  the  JHMCS  JPO  controls  for  updating  the 
acquisition  and  test  plans  are  inadequate.  Acquisition  planning  strategy  and  risk 
assessment  is  clearly  documented  in  JHMCS  documents  other  than  the  SAMP.  Test  plans  are 
living  documents  and  have  been  under  constant  review  and  change  throughout  the  life  of  this 
program.  A  TEMP  revision  to  update  the  TEMP  to  agree  with  current  acquisition  strategy  is  now 
under  review  by  the  JHMCS  IPT.  $17M  savings  due  to  component  breakout  Is  not  achievable 
given  the  IG  report  assumptions.  The  IG  analysis  is  flawed  (reduces  contractor  profit  to  zero) 
and  does  not  consider  additional  costs  to  the  Government  to  manage  additional  contracts, 
additional  GFE,  and  be  responsible  for  system  integration. 


Revised 


Revised 


Additional  Commerrts  and  Corrections 

The  following  comments  are  provided  concerning  the  “Executive  Summary"  and  “Background." 

It  appears  the  DoD  IG  Team  does  not  have  a  dear  understanding  of  the  JHMCS.  The  IG  began 
their  introduction  by  stating: 

"The  Joint  Helmet  Mounted  Cueing  System  consists  of  a  helmet  mounted  display 
unit  and  aircraft  interface  components  that  will  allow  aircraft  fighter  pilots  to 
engage  and  destroy  airborne  targets  within  visual  range  with  a  first  look,  first 
shot,  first  loll  advantage  provided  by  the  High-Off-Boresight  Capability.  The 
capability  allows  the  pitots  to  engage,  lock,  and  launch  weapons  at  a  target 
beyond  the  range  of  the  aircraft’s  radar  and  beyond  the  piiof  s  field  of  view.  The 
capability  works  with  the  Navy’s  AIM>9X  misslie.” 

•  While  references  to  "first  kill"  appeared  in  early  program  documents  (e.g.,  Joint 
Mission  Need  Statement,  14  Feb  1995),  this  capability  can  not  be  provided  by 
JHMCS  alone  and  has  been  deleted  from  the  current  ORD,  TEMP,  and  Test  Plans. 
JHMCS  Is  primarily  a  cueing  and  display  system  that  significantly  contributes  to  first- 
shot  HOBS  capability  only  when  integrated  with  the  pilot,  aircraft,  sensors,  and  an 
appropriate  HOBS  weapon,  such  as  AIM-9X. 

•  It  appears  the  IG  does  not  understand  the  employment  of  the  AIM-9X  or  the 
limitations  of  JHMCS  in  the  WVR  arena.  The  report  states:  “The  (HOBS) 
capability  allows  the  pilots  to  engage,  lock,  and  launch  weapons  at  a  target 
beyond  the  range  of  the  aircraft’s  radar  and  beyond  the  pilot’s  field  of  view.” 
This  statement  is  not  accurate.  Although  JHMCS  can  cue  a  HOBS  weapon  beyond 
radar  field-of-regard  (gimbal  limits  of  the  radar),  radar  range  is  basically  a  function  of 
distance,  which  is  unrelated  to  HOBS  cueing.  Also,  for  a  pilot  to  visually  cue  radar  or 
weapons  to  a  target  fiie  target  must  be  within,  not  beyond,  their  field  of  regard.  The 
JHMCS  is  capable  of  providing  cueing  Information  anywhere  the  pilot  looks.  The 
only  limitation  is  the  capability  of  specific  sensors  to  follow  the  pilot  cue.  Many 
sensors  have  gimbal  locks  that  preclude  completely  follovwng  the  pitot  cue.  As 
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sensors  improve  and  are  upgraded,  the  JHMCS  will  be  able  to  support  this  upgraded 
capability. 

The  IG  report  states  that  the  HOBS  capability  “works  with  the  Navy’s  AIM-9X 

missile.”  This  statement  should  be  changed.  Unlike  some  Mriier  service-specrfic 

versions  of  the  Sidewinder  missile,  the  AIM-9X  is  a  MS 
Navy  as  lead  service  for  development  JHMCS  use  is  not  limrted  to  *e  AIM-9X. 
JHMCS  can  be  used  with  a  variety  of  inventory  weapons  and  any  other 
sensor,  including  air-to-ground,  the  vrarfighter  chooses  to  integrate  with  JHMCS. 

The  IG  report  claims  "the  absence  of  minimum  threat  and  objective  valuw  in  the 
ORD  and  the  TEMP”  as  one  of  the  reasons  the  testers  plan  to  ^e  fnendly  arcraft 
data  as  a  comparative  measure  of  operational  effectiveness  and  suitability.  This 
statement  is  both  inaccurate  and  misleading.  The  report  does  not  recognize 

AFOTEC's  attempts  to  schedule  realistic  threat  adversaries,  nor  does  it  rec^nize 
previous  missions  that  were  flown  against  such  adversaries.  As  a 
linstation,  the  test  plan  and  the  TEMP  clearly  state  that  the  use  of  surrogate  and/or 
actual  threat  aircr^  is  a  function  of  availability. 

The  $641  milHon  identified  as  estimated  total  prograrn  cost  is  inwrrect  **  F;® 

cost  for  developing  and  producing  1,776  helmets  and  associated  aircraft  modification 

kits. 


Final  Report 
Reference 


Revised 


Revised 
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OEPARTMENTOFTHE  AIRFORCE 
MR  FORCE  OPERAHONM.  TEST  MO  CMUIMIQN  CS^TBl  (AFOTEC) 
MROMID  MR  FORCE  BASE.  NM  S7117-S66B 


memorandum  FOR:  ASSISTANT  WSPECTOR  GENERAL  FOR  AUDITING 
OFFICE  OF  THE  INSPECTOR  GENERAL 
DEPARTMENT  OF  DEFENSE 


12  Feb  2001 


FROM:  AFOTEC/CV 

8500  6BS0N  BLVD  SE 
KIRTLAND  AFB,  NM  871174558 


SUBJECT:  DOD  IG  Draft  Joint  Helmot  Mounted  Cueing  System  (tftMCS)  Audit  Report,  Dated 
December  8, 2000 


1.  The  following  Information  is  provided  in  response  to  the  subject  DOD  IG  draft  audit  report. 
7heiGtean«i*dfci»»wportlwt“The  JoW  Helnwt  Mounted  Cueing  System 
operational  tests  wHI  not  provMs  the  ohjective  test  results  necessary  to  support  the  full- 
rate  production  dedsion  in  2002.” 


2.  The  draft  report  nsted  the  following  issues^concems  with  the  MuRHServIce  OperaBo^  Test 
aid  Evaluaeon  (M0T8Q  Ptei,  date!  October  1999.  "The  Multi-Service  Operational  Test  and 
Evaluation  Plan  did  not  Include  a  baseline  threshold  or  ol^isctives  to  measure  success, 
did  not  include  plans  for  a  valid  statistical  projection,  did  not  specify  pass-anctfoil 
crit^  and  did  not  include  a  confidence  level  for  questionnaires  developed  to  measure 
and  project  human  factor  elements  of  critical  operational  issues.” 


Response:  Non-Concur 

Discussion:  The  baseline  eftecCveness  measures  are  to  be  evaluated  by  flying  comparison 
missions  of  F-IS/FA-IS  aircfaft  with  JHMCS  vs.  F-IS/FA-IS  wthout  JHMCS.  The  test  plan 
isted  avalt^llty  of  actual  threat  aircraft  with  a  helmet  cueing  system  as  a  llmllatlon  to  the  test 
K  also  states  that  flights  will  be  conducted  against  the  actual  threat  aircraft  If  available.  The  IG 
stated  ttiat  as  a  minimum  a  baseline  comparison  should  be  accomplished  vs.  the  toreat  aircraft. 
The  test  team  win  conduct  a  baseHne  comparison  of  threshold  aircraft  with  a  JHMCS  system 
and  ABd-OX  versus  known  threat  aircraft  capabilities  as  ouflined  In  the  Detailed  Test  Procedures 
and  Date  Management  and  Analysis  Plan.  The  updated  Test  and  Evaluation  Master  Plan 
(TEMP)  will  Include  threat  capabilities  to  be  used  for  the  baseline  comparison. 


The  Induslon  of  pass/faB  criteria  and  confidence  levels  for  the  subjective  human  factors  data 
would  not  be  stetisticaily  vafld  given  the  smafi  sample  of  12  Air  Force  and  Navy  test  pitots  that 
will  be  ufliized  during  MOT&E.  A  valid  stefistical  projection  plan  would  require  using  a  sample 
size  of  helmets  and  pilots  that  would  far  exceed  the  combined  resources  of  both  the  Air  Force 
and  Navy  operational  test  agencies.  However,  the  current  sample  of  pilots  Is  suffiderrt  to 
provide  dear  answers  concerning  the  impact  of  human  factors  elements  on  JHMCS  operational 
effectiveness  and  sultabtilty.  A  more  detailed  explanation  of  the  AFOTEC  position  on  these 
issues  may  be  found  In  Attachment  1. 

3.  One  of  •»  report  recornnandaiiDns  wee:  “We  recommend  (hat  tire  Director  (^), 

Force  Op^onal  Te$t  and  Evahiatlon  Force  (sic),  revise  the  Multi-Service  Operational 


Iffacy.  SoBg^iess  Future..  ToBr  Natioii's  Air  Force 
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T« tt  and  Evaluation  Wan  to  includa  changaa  mada  to  th#  and  Evaluation  Hai^ 
Plan  that  alM  tha  Plan,  idantify  pass  and  fail  criteria  for  tha  quastionnairas  usad  to 
avaluate  human  factors,  and  revIsa  tha  statistical  projactlon  plan  ” 

Rasponsa:  Mon-concur 

We  vM  address  any  TByiP  updates  that  affect  the  MOT&E  We  do  not  (pa^wilh^ 
fBoornmendation  to  identify  pesa/faii  cfHeHa  for  Bie  human  fiactore  (jiiestionnwas.  Tne 
questionnaires  are  intended  to  galher  afojecfive  hputeand  wfll 
the  operational  teipacte  of  supporting  and  empioj^  this  new  capabify.  As  stetM  m  ire 
Plan/giSOTrtake  results  wIB  be 

observations,  Interviews,  mission  debrSET  and  deficaency  ^ 

that  wil  atow  us  to  render  a  professional  opinion  on  the  >IMCS  operational  effectiveness  and 

suitaU&ty. 

4.  The  JHMCS  MOT&E  wil  provide  the  necessary  data  to  a^jport  a  JHMCS  foliate  production 
decision  in  April  2002. 


CHMRffiS  W.  OHUmN  ^  ^ 


CHMRffiSW. 
Colonel,  USAF 
Vice  Commander 


Attachment 

Memorandum  for  Record 
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MEMORANDUM  FOR  THE  RECORD 


30  January  2001 


SUBJECT;  Draft  DOO  IS  Audft  Report  on  the  Acquisition  of  the  Joint  Helmei 
Moixted  Cueing  System  (JHMCS):  Human  Factors  Methodology  Isaues 

1.  The  draft  DOD  IG  audft  report  on  the  Acqufeaon  rf  the  JHMCS  raead  two  issues 

fiat  pertain  dbectty  to  the  evaiualion  of  JHMCS  human  toctors.  Fkst»  the  IG  report 
stated  that  the  smal,  non-random,  sample  of  ^  Air  Force  and  six  Navy  pilots  is 
imdequate  to  achiewe  staHstirat  oonfidenoe  and  ‘^aocuratoty  project  the  operalioitti 
oftpcftwenesB  of  human  lectors  to  the  uniwarae  of  Air  Force  and  Navy  piotsT.  Second, 
tie  IG  has  stated  ttiat  the  lack  of  quanUfiable  pass/bl  criteria  tir  the  subjective, 
queettomaire  baMd,  of  human  fadors  vM  not  provide  a  dear  answer 

regardng  tie  elfecGveneae  and  suftabHy  of  JHMCS  hunan  tectais.  Theee  two  issues 
are  dosely  iilenneialed  and  must  be  addressed  jcWly. 

2.  The  DOD-IG  indicatod  that,  ‘Wnoe  each  test  plWs  hteracfion  wilh  the  JFWCS  is 
todependert,  staBsScal  sarrping  methods  must  be  used  to  accuraWy  represent  the 
hetoiefs  operatkmd  etiedivenese.*  Although  this  is  e  reasoneble  assumption,  the  costs 
tyfifrratml  wHh  achfevtog  statisticsf  oonfidenoe  must  be  considered.  For  example, 

a  pQpulalion  of  1000  helmals,  adiieving  a  samplng  error  of  plus  or  mirus  five  percent 
wtti  a  confidence  interval  of  80%  requres  a  sample  size  of  14a  Increashg  the 
sampling  error  to  plus  or  mirius  10  percert  reduces  tie  reqdied  sample  to  40.  The 
sufaetenlid  rods  assodated  witi  procuring  adcffionat  test  hetmete  as  wel  as  the 
tremendous  hcreaee  In  tie  number  of  required  lest  flights  to  achieve  even  a  modest 
de^ee  of  stalisfical  oonfldence  would  be  prohftiiiive  givan  the  current  test  budget 

3.  The  IG  audi  alao  questioned  the  non^andom  selection  of  pflols  participating  to  tie 
operational  tset  From  a  statisScal  viewpoint  random  samping  b  neoeesary  in  order  to 
generaizB  the  resufls  itom  a  sample  to  a  subject  popuia^  hcMrever,  random  sampfing 
b  not  reasondrie  or  practice  in  tie  operational  test  environment  It  would  impose  a 
tremendous  burden  on  active  mfliary  unis  to  provide  randomly  selected  piiola  to 
partidpato  to  operettonai  tests  for  an  extended  period  of  time.  AddSonafly,  it  should  be 
noted  that  the  picte  seiectod  to  evaluate  JHMCS  are  drawn  from  a  pod  of  experts  wflh 
extensive  experience  in  detenrintog  tie  mMaiy  ulity  of  the  systems  under  test  Prior 
operaSonal  testing  eoqperierxre  (as  wed  as  tie  narahire  tom  oommerdal  software 
tasting)  has  shown  that  a  smai  group  of  e9<perts  b  capable  of  ktenWytog  the  vast 
iTtejori^  of  sitostanOve  system  errors.  Although  ft  b  dear  teat  random  sampling  and 
Increasing  the  sample  size  would  increase  stafisticai  oonfidenoe  in  tie  resuHs,  it  seems 
that  the  costs  toherent  In  obtaining  a  datisticaly  valid  sample  would  for  atwslgh  the 
benefXs. 

4.  The  absence  of  pass/Edi  criteria  for  tie  human  foctots  data  b  driven  to  a  large  extent 
by  foe  smdl  san^  size  that  Is  typically  available  for  evaluating  operational  qrstems. 

As  noted  previously,  JHMCS  would  require  a  minimum  sample  of  40  pads  in  order  to 
asdgn  an  objective  pasaffol  criteria  wih  any  degree  of  statistical  confidence  and  ft 
would  be  statistically  invalid  to  apply  criteria  to  date  based  on  a  small  sample. 

Furfoemiore,  evan  If  ai  adequate  sample  were  awailabte.  there  are  several  dher  bsues 
that  shodd  be  considered  before  pess^T  criteria  oouid  be  assigned.  Human  foctors 
evaluations  are  inherently  subjective  because  objective  criteria  are  not  a\rai!abie  for  most 
of  foe  issues  under  consideration  (e.g.,  helmet  comfort  or  situational  awarer»ss). 
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the  subiecfive.  questtorviaire  based  data  used  to  answer  these  questions  are  typically 
nonW^  which  limits  the  appIcabiHy  of 

addfiw,  there  is  no  sdentilic  or  operationAy  jusflfied  basts  for  accepting  a  parli^ 
subiecSve  ratng  as  a  criterion  for  a  system  meetiM  an  operaSwal  re^jeme^ 
nStrf  judgment,  efoich  should  be  made,  based  upon  Wh  (he  qye^kxv^ 
and  additional  sources  of  data  received  from  toe  test  subjects,  including  wntten  and  ora! 
comments. 

5.  In  eonsidetalion  of  the  above  issues,  »  was  ^ 

oUadive  pasa/bl  criteria  to  the  human  factors  analyses  of  JHMCS  {or  mKi  cU^ 
operafionsIlfaS^ptogwns).  Howeusf,  a  deWtate  anaaer  oonoemng  ^  aUedivenass 
aid  sutebite  of  JHMCS  human  factors  wil  be  piwi^  For  a^  subjoc^ 
MO^riioP/Soi  toe  ^stem  wil  be  nied  as  favorable  i  toe  mai^  «» 
connente  are  poafiva  or  unfavorable  If  toe  msjoi^  of  toe  ’***’9®  *® 

unfavorable.  Addfional  interviews  vril  be  conducted  i"  fglancesw^  ^ 

andfer  commente  do  not  provide  a  dear  answer.  In  ar^,  ^ 

used  to  sunvnarizB  toe  date  and  a  nanalive  summary  of  toe  ma)or  soengtos  ana 
weaknesses  of  toe  system  wil  be  provided. 
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